Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-16805

Add PipelineTask support for optional input datasets

    XMLWordPrintable

Details

    • Data Release Production

    Description

      The straylight correction for HSC does not clearly follow the same coding structure, correction file, or design as other ISR methods.  This should probably be updated to work more like other corrections.

      This (and fringe support) can be addressed by adding some kind of optional input support to PipelineTask, indicating that zero instances of an input dataset are allowable.

      Attachments

        Issue Links

          Activity

            No builds found.
            czw Christopher Waters created issue -
            jbosch Jim Bosch made changes -
            Field Original Value New Value
            Link This issue relates to DM-19393 [ DM-19393 ]
            jbosch Jim Bosch added a comment -

            After DM-19393, I think the approach here should look something like this:

            • Make preflight query support null results for some dataset types; this probably means we need a way for a PipelineTask to indicate that an input dataset type is optional.  Is that all we need to fix fringes as well?
            • Add a Gen3 Formatter implementation to obs_subaru for the new StrayLightData class, which is "loaded" by just passing the filename to the contructor.
            jbosch Jim Bosch added a comment - After DM-19393 , I think the approach here should look something like this: Make preflight query support null results for some dataset types; this probably means we need a way for a PipelineTask to indicate that an input dataset type is optional.  Is that all we need to fix fringes as well? Add a Gen3 Formatter implementation to obs_subaru for the new StrayLightData class, which is "loaded" by just passing the filename to the contructor.

            I believe that will fix the simple fringe case as well.  If this becomes just a simple boolean (`isOptional`) in `InputDatasetField`, will this cause conflict with the `scalar=False` option?  The end case for fringes is that we would want to support a NULL result (for filter != y), a single entry (for a simple fringe), and a list (for a complex multi-mode fringe).

            czw Christopher Waters added a comment - I believe that will fix the simple fringe case as well.  If this becomes just a simple boolean (`isOptional`) in `InputDatasetField`, will this cause conflict with the `scalar=False` option?  The end case for fringes is that we would want to support a NULL result (for filter != y), a single entry (for a simple fringe), and a list (for a complex multi-mode fringe).
            jbosch Jim Bosch added a comment -

            Good question; I'm starting to think we might want to replace the boolean scalar with some kind of multiple-choice field.  It's now feeling a lot like that narg options in Python's argparse module (though without perhaps quite so many options).

            jbosch Jim Bosch added a comment - Good question; I'm starting to think we might want to replace the boolean scalar with some kind of multiple-choice field.  It's now feeling a lot like that narg options in Python's argparse module (though without perhaps quite so many options).
            jbosch Jim Bosch made changes -
            Assignee Jim Bosch [ jbosch ]
            jbosch Jim Bosch made changes -
            Summary obs_subaru straylight correction doesn't match other ISR methods Add PipelineTask support for optional input datasets
            jbosch Jim Bosch made changes -
            Description The straylight correction for HSC does not clearly follow the same coding structure, correction file, or design as other ISR methods.  This should probably be updated to work more like other corrections. The straylight correction for HSC does not clearly follow the same coding structure, correction file, or design as other ISR methods.  This should probably be updated to work more like other corrections.

            This (and fringe support) can be addressed by adding some kind of optional input support to PipelineTask, indicating that zero instances of an input dataset are allowable.
            jbosch Jim Bosch made changes -
            Labels gen3-middleware
            jbosch Jim Bosch made changes -
            Link This issue relates to DM-17152 [ DM-17152 ]
            jbosch Jim Bosch made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            jbosch Jim Bosch made changes -
            Remote Link This issue links to "Page (Confluence)" [ 20819 ]
            jbosch Jim Bosch added a comment -

            I think the base PipelineTask functionality for this was completed on DM-20205; what remains is specific to IsrTask, and I'll open a new ticket for that.

            jbosch Jim Bosch added a comment - I think the base PipelineTask functionality for this was completed on DM-20205 ; what remains is specific to IsrTask, and I'll open a new ticket for that.
            jbosch Jim Bosch made changes -
            Link This issue is contained by DM-20205 [ DM-20205 ]
            jbosch Jim Bosch made changes -
            Status In Progress [ 3 ] To Do [ 10001 ]
            jbosch Jim Bosch made changes -
            Resolution Done [ 10000 ]
            Status To Do [ 10001 ] Invalid [ 11005 ]

            People

              jbosch Jim Bosch
              czw Christopher Waters
              Christopher Waters, Jim Bosch, Krzysztof Findeisen
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.