Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-16805

Add PipelineTask support for optional input datasets

    XMLWordPrintable

    Details

    • Team:
      Data Release Production

      Description

      The straylight correction for HSC does not clearly follow the same coding structure, correction file, or design as other ISR methods.  This should probably be updated to work more like other corrections.

      This (and fringe support) can be addressed by adding some kind of optional input support to PipelineTask, indicating that zero instances of an input dataset are allowable.

        Attachments

          Issue Links

          is contained by

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-20205 Refactor PipelineTask interface

          • Done
          relates to

          Improvement - An improvement or enhancement to an existing feature or task. DM-17152 Fringe dataset does not easily fit the current Gen3 butler concept

          • Invalid

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-19393 Fix HSC y stray-light lookup In Gen2

          • Done
          mentioned in

          Page Loading...

            Activity

            Ascending order - Click to sort in descending order
            Hide
            Permalink
            jbosch Jim Bosch added a comment -

            After DM-19393, I think the approach here should look something like this:

            • Make preflight query support null results for some dataset types; this probably means we need a way for a PipelineTask to indicate that an input dataset type is optional.  Is that all we need to fix fringes as well?
            • Add a Gen3 Formatter implementation to obs_subaru for the new StrayLightData class, which is "loaded" by just passing the filename to the contructor.
            Show
            jbosch Jim Bosch added a comment - After DM-19393 , I think the approach here should look something like this: Make preflight query support null results for some dataset types; this probably means we need a way for a PipelineTask to indicate that an input dataset type is optional.  Is that all we need to fix fringes as well? Add a Gen3 Formatter implementation to obs_subaru for the new StrayLightData class, which is "loaded" by just passing the filename to the contructor.
            Hide
            Permalink
            czw Christopher Waters added a comment -

            I believe that will fix the simple fringe case as well.  If this becomes just a simple boolean (`isOptional`) in `InputDatasetField`, will this cause conflict with the `scalar=False` option?  The end case for fringes is that we would want to support a NULL result (for filter != y), a single entry (for a simple fringe), and a list (for a complex multi-mode fringe).

            Show
            czw Christopher Waters added a comment - I believe that will fix the simple fringe case as well.  If this becomes just a simple boolean (`isOptional`) in `InputDatasetField`, will this cause conflict with the `scalar=False` option?  The end case for fringes is that we would want to support a NULL result (for filter != y), a single entry (for a simple fringe), and a list (for a complex multi-mode fringe).
            Hide
            Permalink
            jbosch Jim Bosch added a comment -

            Good question; I'm starting to think we might want to replace the boolean scalar with some kind of multiple-choice field.  It's now feeling a lot like that narg options in Python's argparse module (though without perhaps quite so many options).

            Show
            jbosch Jim Bosch added a comment - Good question; I'm starting to think we might want to replace the boolean scalar with some kind of multiple-choice field.  It's now feeling a lot like that narg options in Python's argparse module (though without perhaps quite so many options).
            Hide
            Permalink
            jbosch Jim Bosch added a comment -

            I think the base PipelineTask functionality for this was completed on DM-20205; what remains is specific to IsrTask, and I'll open a new ticket for that.

            Show
            jbosch Jim Bosch added a comment - I think the base PipelineTask functionality for this was completed on DM-20205 ; what remains is specific to IsrTask, and I'll open a new ticket for that.

              People

              Assignee:
              jbosch Jim Bosch
              Reporter:
              czw Christopher Waters
              Watchers:
              Christopher Waters, Jim Bosch, Krzysztof Findeisen
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.