Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-16805

Add PipelineTask support for optional input datasets

    XMLWordPrintable

Details

    • Data Release Production

    Description

      The straylight correction for HSC does not clearly follow the same coding structure, correction file, or design as other ISR methods.  This should probably be updated to work more like other corrections.

      This (and fringe support) can be addressed by adding some kind of optional input support to PipelineTask, indicating that zero instances of an input dataset are allowable.

      Attachments

        Issue Links

          Activity

            jbosch Jim Bosch added a comment -

            After DM-19393, I think the approach here should look something like this:

            • Make preflight query support null results for some dataset types; this probably means we need a way for a PipelineTask to indicate that an input dataset type is optional.  Is that all we need to fix fringes as well?
            • Add a Gen3 Formatter implementation to obs_subaru for the new StrayLightData class, which is "loaded" by just passing the filename to the contructor.
            jbosch Jim Bosch added a comment - After DM-19393 , I think the approach here should look something like this: Make preflight query support null results for some dataset types; this probably means we need a way for a PipelineTask to indicate that an input dataset type is optional.  Is that all we need to fix fringes as well? Add a Gen3 Formatter implementation to obs_subaru for the new StrayLightData class, which is "loaded" by just passing the filename to the contructor.

            I believe that will fix the simple fringe case as well.  If this becomes just a simple boolean (`isOptional`) in `InputDatasetField`, will this cause conflict with the `scalar=False` option?  The end case for fringes is that we would want to support a NULL result (for filter != y), a single entry (for a simple fringe), and a list (for a complex multi-mode fringe).

            czw Christopher Waters added a comment - I believe that will fix the simple fringe case as well.  If this becomes just a simple boolean (`isOptional`) in `InputDatasetField`, will this cause conflict with the `scalar=False` option?  The end case for fringes is that we would want to support a NULL result (for filter != y), a single entry (for a simple fringe), and a list (for a complex multi-mode fringe).
            jbosch Jim Bosch added a comment -

            Good question; I'm starting to think we might want to replace the boolean scalar with some kind of multiple-choice field.  It's now feeling a lot like that narg options in Python's argparse module (though without perhaps quite so many options).

            jbosch Jim Bosch added a comment - Good question; I'm starting to think we might want to replace the boolean scalar with some kind of multiple-choice field.  It's now feeling a lot like that narg options in Python's argparse module (though without perhaps quite so many options).
            jbosch Jim Bosch added a comment -

            I think the base PipelineTask functionality for this was completed on DM-20205; what remains is specific to IsrTask, and I'll open a new ticket for that.

            jbosch Jim Bosch added a comment - I think the base PipelineTask functionality for this was completed on DM-20205 ; what remains is specific to IsrTask, and I'll open a new ticket for that.

            People

              jbosch Jim Bosch
              czw Christopher Waters
              Christopher Waters, Jim Bosch, Krzysztof Findeisen
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.