Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-4077

Support external processing inputs in mapper and argument parser

    Details

    • Type: Story
    • Status: Invalid
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: obs_base, pipe_base
    • Labels:
      None
    • Team:
      Data Access and Database

      Description

      RFC-95 proposes new features in the Butler (probably actually implemented in CameraMapper) and the CmdLineTask argument parser to support running the pipeline on intermediate data products produced by external pipeline, such as the DECam community pipeline.

      This issue represents the implementation of that feature.

        Attachments

          Issue Links

            Activity

            Hide
            ktl Kian-Tat Lim added a comment -

            I'm not sure what from RFC-95 requires Butler changes. At first glance, it all looked like pipe_base stuff, with a few more dataId components set by command line arguments. Can you clarify?

            Show
            ktl Kian-Tat Lim added a comment - I'm not sure what from RFC-95 requires Butler changes. At first glance, it all looked like pipe_base stuff, with a few more dataId components set by command line arguments. Can you clarify?
            Hide
            jbosch Jim Bosch added a comment -

            I was thinking that we needed something much like the separate calibration registry to support a choice of external input data, and my understanding was that this was in CameraMapper, not pipe_base. But I also don't think the proposal in the RFC absolutely requires that; I tried to describe the user features that were required there but I left the implementation somewhat vague intentionally.

            Show
            jbosch Jim Bosch added a comment - I was thinking that we needed something much like the separate calibration registry to support a choice of external input data, and my understanding was that this was in CameraMapper, not pipe_base. But I also don't think the proposal in the RFC absolutely requires that; I tried to describe the user features that were required there but I left the implementation somewhat vague intentionally.
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            Is ProcessCcdTask or its camera-specific subclass the only step that would take preprocessed data? In other words, is the only preprocessed data we care postISR-equivalent product?
            If so, once they are processed pass ProcessCcd, users don't distinguish whether data were originated from raw or from preprocessed data? And "preprocessed" is an argument only wanted in ProcessCcdTask, but not other tasks?

            Show
            hchiang2 Hsin-Fang Chiang added a comment - Is ProcessCcdTask or its camera-specific subclass the only step that would take preprocessed data? In other words, is the only preprocessed data we care postISR-equivalent product? If so, once they are processed pass ProcessCcd, users don't distinguish whether data were originated from raw or from preprocessed data? And "preprocessed" is an argument only wanted in ProcessCcdTask , but not other tasks?
            Hide
            jbosch Jim Bosch added a comment -

            Is ProcessCcdTask or its camera-specific subclass the only step that would take preprocessed data? In other words, is the only preprocessed data we care postISR-equivalent product?
            If so, once they are processed pass ProcessCcd, users don't distinguish whether data were originated from raw or from preprocessed data? And "preprocessed" is an argument only wanted in ProcessCcdTask, but not other tasks?

            We may someday have different kinds of preprocessed data other than post-ISR products, such as coadds built with an entirely different pipeline. Those may be different enough that we shouldn't spend effort trying to include them in the design now, though, so it may best to just assume that only ProcessCcdTask needs to worry about preprocessed data and worry about the future in the future.

            Show
            jbosch Jim Bosch added a comment - Is ProcessCcdTask or its camera-specific subclass the only step that would take preprocessed data? In other words, is the only preprocessed data we care postISR-equivalent product? If so, once they are processed pass ProcessCcd, users don't distinguish whether data were originated from raw or from preprocessed data? And "preprocessed" is an argument only wanted in ProcessCcdTask , but not other tasks? We may someday have different kinds of preprocessed data other than post-ISR products, such as coadds built with an entirely different pipeline. Those may be different enough that we shouldn't spend effort trying to include them in the design now, though, so it may best to just assume that only ProcessCcdTask needs to worry about preprocessed data and worry about the future in the future.
            Hide
            fritzm Fritz Mueller added a comment -

            No longer relevant. RFC-95 was withdrawn, and per Kian-Tat Lim, DECam CP data is already being used.

            Show
            fritzm Fritz Mueller added a comment - No longer relevant. RFC-95 was withdrawn, and per Kian-Tat Lim , DECam CP data is already being used.

              People

              • Assignee:
                npease Nate Pease
                Reporter:
                jbosch Jim Bosch
                Watchers:
                Fritz Mueller, Hsin-Fang Chiang, Jim Bosch, Kian-Tat Lim
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel