Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-16805

Add PipelineTask support for optional input datasets

    XMLWordPrintable

    Details

    • Team:
      Data Release Production

      Description

      The straylight correction for HSC does not clearly follow the same coding structure, correction file, or design as other ISR methods.  This should probably be updated to work more like other corrections.

      This (and fringe support) can be addressed by adding some kind of optional input support to PipelineTask, indicating that zero instances of an input dataset are allowable.

        Attachments

          Issue Links

            Activity

            No builds found.
            czw Christopher Waters created issue -
            jbosch Jim Bosch made changes -
            Field Original Value New Value
            Link This issue relates to DM-19393 [ DM-19393 ]
            Hide
            jbosch Jim Bosch added a comment -

            After DM-19393, I think the approach here should look something like this:

            • Make preflight query support null results for some dataset types; this probably means we need a way for a PipelineTask to indicate that an input dataset type is optional.  Is that all we need to fix fringes as well?
            • Add a Gen3 Formatter implementation to obs_subaru for the new StrayLightData class, which is "loaded" by just passing the filename to the contructor.
            Show
            jbosch Jim Bosch added a comment - After DM-19393 , I think the approach here should look something like this: Make preflight query support null results for some dataset types; this probably means we need a way for a PipelineTask to indicate that an input dataset type is optional.  Is that all we need to fix fringes as well? Add a Gen3 Formatter implementation to obs_subaru for the new StrayLightData class, which is "loaded" by just passing the filename to the contructor.
            Hide
            czw Christopher Waters added a comment -

            I believe that will fix the simple fringe case as well.  If this becomes just a simple boolean (`isOptional`) in `InputDatasetField`, will this cause conflict with the `scalar=False` option?  The end case for fringes is that we would want to support a NULL result (for filter != y), a single entry (for a simple fringe), and a list (for a complex multi-mode fringe).

            Show
            czw Christopher Waters added a comment - I believe that will fix the simple fringe case as well.  If this becomes just a simple boolean (`isOptional`) in `InputDatasetField`, will this cause conflict with the `scalar=False` option?  The end case for fringes is that we would want to support a NULL result (for filter != y), a single entry (for a simple fringe), and a list (for a complex multi-mode fringe).
            Hide
            jbosch Jim Bosch added a comment -

            Good question; I'm starting to think we might want to replace the boolean scalar with some kind of multiple-choice field.  It's now feeling a lot like that narg options in Python's argparse module (though without perhaps quite so many options).

            Show
            jbosch Jim Bosch added a comment - Good question; I'm starting to think we might want to replace the boolean scalar with some kind of multiple-choice field.  It's now feeling a lot like that narg options in Python's argparse module (though without perhaps quite so many options).
            jbosch Jim Bosch made changes -
            Assignee Jim Bosch [ jbosch ]
            jbosch Jim Bosch made changes -
            Summary obs_subaru straylight correction doesn't match other ISR methods Add PipelineTask support for optional input datasets
            jbosch Jim Bosch made changes -
            Description The straylight correction for HSC does not clearly follow the same coding structure, correction file, or design as other ISR methods.  This should probably be updated to work more like other corrections. The straylight correction for HSC does not clearly follow the same coding structure, correction file, or design as other ISR methods.  This should probably be updated to work more like other corrections.

            This (and fringe support) can be addressed by adding some kind of optional input support to PipelineTask, indicating that zero instances of an input dataset are allowable.
            jbosch Jim Bosch made changes -
            Labels gen3-middleware
            jbosch Jim Bosch made changes -
            Link This issue relates to DM-17152 [ DM-17152 ]
            jbosch Jim Bosch made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            jbosch Jim Bosch made changes -
            Remote Link This issue links to "Page (Confluence)" [ 20819 ]
            Hide
            jbosch Jim Bosch added a comment -

            I think the base PipelineTask functionality for this was completed on DM-20205; what remains is specific to IsrTask, and I'll open a new ticket for that.

            Show
            jbosch Jim Bosch added a comment - I think the base PipelineTask functionality for this was completed on DM-20205 ; what remains is specific to IsrTask, and I'll open a new ticket for that.
            jbosch Jim Bosch made changes -
            Link This issue is contained by DM-20205 [ DM-20205 ]
            jbosch Jim Bosch made changes -
            Status In Progress [ 3 ] To Do [ 10001 ]
            jbosch Jim Bosch made changes -
            Resolution Done [ 10000 ]
            Status To Do [ 10001 ] Invalid [ 11005 ]

              People

              Assignee:
              jbosch Jim Bosch
              Reporter:
              czw Christopher Waters
              Watchers:
              Christopher Waters, Jim Bosch, Krzysztof Findeisen
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  CI Builds

                  No builds found.