Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-16805

Add PipelineTask support for optional input datasets

    Details

    • Team:
      Data Release Production

      Description

      The straylight correction for HSC does not clearly follow the same coding structure, correction file, or design as other ISR methods.  This should probably be updated to work more like other corrections.

      This (and fringe support) can be addressed by adding some kind of optional input support to PipelineTask, indicating that zero instances of an input dataset are allowable.

        Attachments

          Issue Links

            Activity

            Hide
            jbosch Jim Bosch added a comment -

            After DM-19393, I think the approach here should look something like this:

            • Make preflight query support null results for some dataset types; this probably means we need a way for a PipelineTask to indicate that an input dataset type is optional.  Is that all we need to fix fringes as well?
            • Add a Gen3 Formatter implementation to obs_subaru for the new StrayLightData class, which is "loaded" by just passing the filename to the contructor.
            Show
            jbosch Jim Bosch added a comment - After DM-19393 , I think the approach here should look something like this: Make preflight query support null results for some dataset types; this probably means we need a way for a PipelineTask to indicate that an input dataset type is optional.  Is that all we need to fix fringes as well? Add a Gen3 Formatter implementation to obs_subaru for the new StrayLightData class, which is "loaded" by just passing the filename to the contructor.
            Hide
            czw Christopher Waters added a comment -

            I believe that will fix the simple fringe case as well.  If this becomes just a simple boolean (`isOptional`) in `InputDatasetField`, will this cause conflict with the `scalar=False` option?  The end case for fringes is that we would want to support a NULL result (for filter != y), a single entry (for a simple fringe), and a list (for a complex multi-mode fringe).

            Show
            czw Christopher Waters added a comment - I believe that will fix the simple fringe case as well.  If this becomes just a simple boolean (`isOptional`) in `InputDatasetField`, will this cause conflict with the `scalar=False` option?  The end case for fringes is that we would want to support a NULL result (for filter != y), a single entry (for a simple fringe), and a list (for a complex multi-mode fringe).
            Hide
            jbosch Jim Bosch added a comment -

            Good question; I'm starting to think we might want to replace the boolean scalar with some kind of multiple-choice field.  It's now feeling a lot like that narg options in Python's argparse module (though without perhaps quite so many options).

            Show
            jbosch Jim Bosch added a comment - Good question; I'm starting to think we might want to replace the boolean scalar with some kind of multiple-choice field.  It's now feeling a lot like that narg options in Python's argparse module (though without perhaps quite so many options).
            Hide
            jbosch Jim Bosch added a comment -

            I think the base PipelineTask functionality for this was completed on DM-20205; what remains is specific to IsrTask, and I'll open a new ticket for that.

            Show
            jbosch Jim Bosch added a comment - I think the base PipelineTask functionality for this was completed on DM-20205 ; what remains is specific to IsrTask, and I'll open a new ticket for that.

              People

              • Assignee:
                jbosch Jim Bosch
                Reporter:
                czw Christopher Waters
                Watchers:
                Christopher Waters, Jim Bosch, Krzysztof Findeisen
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel