Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-21862

Extend Gen3 butler support for obs_decam

    Details

    • Story Points:
      8
    • Sprint:
      AP F19-6 (November), AP S20-5 (April)
    • Team:
      Alert Production

      Description

      obs_decam now has basic Gen3 butler support (sufficient for ingest), but we'll need to expand that to get to the point where we can run single-frame processing (IsrTask, CharacterizeImageTask, and CalibrateTask) via Gen3 on it.  That will include, at least:

      • implementing writeCuratedCalibrations in the instrument class;
      • specializing (and modifying, as necessary) the instrument-specific hooks for the gen2to3 tool, at least to the point where we can write scripts like the ones in ci_hsc_gen3/bin.src to make a fully-usable Gen3 repo.

      I'm assigning this to John Parejko and the AP team for now, in the hopes it might be possible for him to finish the work he started with the initial DECam conversion.

      A useful concrete end-point for this task would be a CI package that creates a Gen3 repo containing the HiTS (2015) raws, master calibrations, reference catalogs and runs the aforementioned tasks as a Pipeline.

        Attachments

          Issue Links

            Activity

            Hide
            ebellm Eric Bellm added a comment -

            DECam has inter-CCD crosstalk, which requires resolving https://jira.lsstcorp.org/browse/DM-17169

            Show
            ebellm Eric Bellm added a comment - DECam has inter-CCD crosstalk, which requires resolving https://jira.lsstcorp.org/browse/DM-17169
            Hide
            Parejkoj John Parejko added a comment - - edited

            Here is a proposal that Tim Jenness and I came up with to deal with with raw ingestion:

            Each obs_foo package will have a `RawIngestFooTask` and associated executable. This Task will be a subclass of `obs.base.RawIngestTask` that knows the correct `Instrument`. For obs packages that have multiple instruments (e.g. obs_lsst), we propose a `ConfigField` to select the correct instrument (e.g. "Ts8"->"Ts8Instrument", "Latiss"->"LatissInstrument"). This should solve the problem of how to specify the instrument that goes with a set of raw files.

            A followup question to this is whether these specialized tasks should have a `doCalibrations` option that runs `writeCuratedCalibrations`, or whether we want a separate `CalibIngestFooTask` for that purpose? We will be doing "raw ingest" much more often than "calibration ingest", but the latter is handled by the same `Instrument` class.

            Show
            Parejkoj John Parejko added a comment - - edited Here is a proposal that Tim Jenness and I came up with to deal with with raw ingestion: Each obs_foo package will have a `RawIngestFooTask` and associated executable. This Task will be a subclass of `obs.base.RawIngestTask` that knows the correct `Instrument`. For obs packages that have multiple instruments (e.g. obs_lsst), we propose a `ConfigField` to select the correct instrument (e.g. "Ts8"->"Ts8Instrument", "Latiss"->"LatissInstrument"). This should solve the problem of how to specify the instrument that goes with a set of raw files. A followup question to this is whether these specialized tasks should have a `doCalibrations` option that runs `writeCuratedCalibrations`, or whether we want a separate `CalibIngestFooTask` for that purpose? We will be doing "raw ingest" much more often than "calibration ingest", but the latter is handled by the same `Instrument` class.
            Hide
            jbosch Jim Bosch added a comment -

            I don't think writeCuratedCalibrations belongs in ingest.  If anything, it belongs in a (TBD) script to register the instrument with the repository, but I don't think even that is quite right; there are several different cadences here, and Instrument's API's aren't really well-aligned to them yet (though they work well for testing and development):

            • Register instrument with repository: done exactly once per repository+instrument combination.  Adds "instrument", "detector", and "physical_filter" dimension records.
            • Human-curated calibrations change: call writeCuratedCalibrations into a new collection, test them, bless that new collection as the recommended one.
            • Hardware changes: add new physical_filter entries, maybe add records of some future dimensions (Tim Jenness and I once talked about having a physical_detector, for example).
            • New data is taken: run raw ingest.

             

            What's the motivation for having a per-instrument Task subclass, rather than just per-instrument configuration?  Are we moving abstract methods out of Instrument and into the base Task, or are we just using derived classes as a place to override config variables?

            Show
            jbosch Jim Bosch added a comment - I don't think writeCuratedCalibrations belongs in ingest.  If anything, it belongs in a (TBD) script to register the instrument with the repository, but I don't think even that is quite right; there are several different cadences here, and Instrument's API's aren't really well-aligned to them yet (though they work well for testing and development): Register instrument with repository: done exactly once per repository+instrument combination.  Adds "instrument", "detector", and "physical_filter" dimension records. Human-curated calibrations change: call writeCuratedCalibrations into a new collection, test them, bless that new collection as the recommended one. Hardware changes: add new physical_filter entries, maybe add records of some future dimensions ( Tim Jenness and I once talked about having a physical_detector , for example). New data is taken: run raw ingest.   What's the motivation for having a per-instrument Task subclass, rather than just per-instrument configuration?  Are we moving abstract methods out of Instrument and into the base Task, or are we just using derived classes as a place to override config variables?
            Hide
            tjenness Tim Jenness added a comment -

            Regarding your final comment, is it a question of ingestDecam.py vs ingest.py --config $OBS_DECAM/config/ingest.py? I think we ask people what they prefer.

            A separate command for preparing a butler repo for an instrument is fine with me. This would presumably be allowed to run repeatedly, adding new filters if required, else doing nothing.

            I had not considered changing collection every time curated calibrations are ingested. That seems fine to me since you seem to be implying that there will always be a default collection for calibrations.

            Show
            tjenness Tim Jenness added a comment - Regarding your final comment, is it a question of ingestDecam.py vs ingest.py --config $OBS_DECAM/config/ingest.py ? I think we ask people what they prefer. A separate command for preparing a butler repo for an instrument is fine with me. This would presumably be allowed to run repeatedly, adding new filters if required, else doing nothing. I had not considered changing collection every time curated calibrations are ingested. That seems fine to me since you seem to be implying that there will always be a default collection for calibrations.
            Hide
            jbosch Jim Bosch added a comment -

            Regarding your final comment, is it a question of ingestDecam.py vs ingest.py --config $OBS_DECAM/config/ingest.py? I think we ask people what they prefer.

            for going with what's considered the best UI.  But also note that we can have different scripts for different cameras without having different tasks, if all the scripts are doing is hard-coding a config override.

             

             

            Show
            jbosch Jim Bosch added a comment - Regarding your final comment, is it a question of ingestDecam.py vs ingest.py --config $OBS_DECAM/config/ingest.py ? I think we ask people what they prefer. for going with what's considered the best UI.  But also note that we can have different scripts for different cameras without having different tasks, if all the scripts are doing is hard-coding a config override.    
            Hide
            Parejkoj John Parejko added a comment -

            I am going to close this ticket as "Done": DM-23976 means that defects are being written in gen3 from obs_decam_data, and DM-22655 got us a mostly-working gen2to3 convert script for decam. DM-23616 includes notes about how to run the script for ap_verify_ci_hits2015, and took care of the groundwork of processing decam data with pipetask. The remaining known issues preventing us from knowing whether the conversion is correct and that we can run gen3 single frame processing are being dealt with on DM-23985, DM-23992, and DM-23983.

            Show
            Parejkoj John Parejko added a comment - I am going to close this ticket as "Done": DM-23976 means that defects are being written in gen3 from obs_decam_data, and DM-22655 got us a mostly-working gen2to3 convert script for decam. DM-23616 includes notes about how to run the script for ap_verify_ci_hits2015, and took care of the groundwork of processing decam data with pipetask. The remaining known issues preventing us from knowing whether the conversion is correct and that we can run gen3 single frame processing are being dealt with on DM-23985 , DM-23992 , and DM-23983 .

              People

              • Assignee:
                Parejkoj John Parejko
                Reporter:
                jbosch Jim Bosch
                Watchers:
                Eric Bellm, Heather Kelly, Jim Bosch, John Parejko, John Swinbank, Krzysztof Findeisen, Meredith Rawls, Simon Krughoff, Tim Jenness
              • Votes:
                0 Vote for this issue
                Watchers:
                9 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel