Extend Gen3 butler support for obs_decam

XMLWordPrintable

Details

• Type: Story
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s:
• Labels:
• Story Points:
8
• Sprint:
AP F19-6 (November), AP S20-5 (April)
• Team:

Description

obs_decam now has basic Gen3 butler support (sufficient for ingest), but we'll need to expand that to get to the point where we can run single-frame processing (IsrTask, CharacterizeImageTask, and CalibrateTask) via Gen3 on it.  That will include, at least:

• implementing writeCuratedCalibrations in the instrument class;
• specializing (and modifying, as necessary) the instrument-specific hooks for the gen2to3 tool, at least to the point where we can write scripts like the ones in ci_hsc_gen3/bin.src to make a fully-usable Gen3 repo.

I'm assigning this to John Parejko and the AP team for now, in the hopes it might be possible for him to finish the work he started with the initial DECam conversion.

A useful concrete end-point for this task would be a CI package that creates a Gen3 repo containing the HiTS (2015) raws, master calibrations, reference catalogs and runs the aforementioned tasks as a Pipeline.

Activity

Hide
Eric Bellm added a comment -

DECam has inter-CCD crosstalk, which requires resolving https://jira.lsstcorp.org/browse/DM-17169

Show
Eric Bellm added a comment - DECam has inter-CCD crosstalk, which requires resolving https://jira.lsstcorp.org/browse/DM-17169
Hide
John Parejko added a comment - - edited

Here is a proposal that Tim Jenness and I came up with to deal with with raw ingestion:

Each obs_foo package will have a RawIngestFooTask and associated executable. This Task will be a subclass of obs.base.RawIngestTask that knows the correct Instrument. For obs packages that have multiple instruments (e.g. obs_lsst), we propose a ConfigField to select the correct instrument (e.g. "Ts8"->"Ts8Instrument", "Latiss"->"LatissInstrument"). This should solve the problem of how to specify the instrument that goes with a set of raw files.

A followup question to this is whether these specialized tasks should have a doCalibrations option that runs writeCuratedCalibrations, or whether we want a separate CalibIngestFooTask for that purpose? We will be doing "raw ingest" much more often than "calibration ingest", but the latter is handled by the same Instrument class.

Show
John Parejko added a comment - - edited Here is a proposal that Tim Jenness and I came up with to deal with with raw ingestion: Each obs_foo package will have a RawIngestFooTask and associated executable. This Task will be a subclass of obs.base.RawIngestTask that knows the correct Instrument. For obs packages that have multiple instruments (e.g. obs_lsst), we propose a ConfigField to select the correct instrument (e.g. "Ts8"->"Ts8Instrument", "Latiss"->"LatissInstrument"). This should solve the problem of how to specify the instrument that goes with a set of raw files. A followup question to this is whether these specialized tasks should have a doCalibrations option that runs writeCuratedCalibrations, or whether we want a separate CalibIngestFooTask for that purpose? We will be doing "raw ingest" much more often than "calibration ingest", but the latter is handled by the same Instrument class.
Hide
Jim Bosch added a comment -

I don't think writeCuratedCalibrations belongs in ingest.  If anything, it belongs in a (TBD) script to register the instrument with the repository, but I don't think even that is quite right; there are several different cadences here, and Instrument's API's aren't really well-aligned to them yet (though they work well for testing and development):

• Register instrument with repository: done exactly once per repository+instrument combination.  Adds "instrument", "detector", and "physical_filter" dimension records.
• Human-curated calibrations change: call writeCuratedCalibrations into a new collection, test them, bless that new collection as the recommended one.
• Hardware changes: add new physical_filter entries, maybe add records of some future dimensions (Tim Jenness and I once talked about having a physical_detector, for example).
• New data is taken: run raw ingest.

What's the motivation for having a per-instrument Task subclass, rather than just per-instrument configuration?  Are we moving abstract methods out of Instrument and into the base Task, or are we just using derived classes as a place to override config variables?

Show
Jim Bosch added a comment - I don't think writeCuratedCalibrations belongs in ingest.  If anything, it belongs in a (TBD) script to register the instrument with the repository, but I don't think even that is quite right; there are several different cadences here, and Instrument's API's aren't really well-aligned to them yet (though they work well for testing and development): Register instrument with repository: done exactly once per repository+instrument combination.  Adds "instrument", "detector", and "physical_filter" dimension records. Human-curated calibrations change: call writeCuratedCalibrations into a new collection, test them, bless that new collection as the recommended one. Hardware changes: add new physical_filter entries, maybe add records of some future dimensions ( Tim Jenness and I once talked about having a physical_detector , for example). New data is taken: run raw ingest.   What's the motivation for having a per-instrument Task subclass, rather than just per-instrument configuration?  Are we moving abstract methods out of Instrument and into the base Task, or are we just using derived classes as a place to override config variables?
Hide
Tim Jenness added a comment -

Regarding your final comment, is it a question of ingestDecam.py vs ingest.py --config $OBS_DECAM/config/ingest.py? I think we ask people what they prefer. A separate command for preparing a butler repo for an instrument is fine with me. This would presumably be allowed to run repeatedly, adding new filters if required, else doing nothing. I had not considered changing collection every time curated calibrations are ingested. That seems fine to me since you seem to be implying that there will always be a default collection for calibrations. Show Tim Jenness added a comment - Regarding your final comment, is it a question of ingestDecam.py vs ingest.py --config$OBS_DECAM/config/ingest.py ? I think we ask people what they prefer. A separate command for preparing a butler repo for an instrument is fine with me. This would presumably be allowed to run repeatedly, adding new filters if required, else doing nothing. I had not considered changing collection every time curated calibrations are ingested. That seems fine to me since you seem to be implying that there will always be a default collection for calibrations.
Hide
Jim Bosch added a comment -

Regarding your final comment, is it a question of ingestDecam.py vs ingest.py --config $OBS_DECAM/config/ingest.py? I think we ask people what they prefer. for going with what's considered the best UI. But also note that we can have different scripts for different cameras without having different tasks, if all the scripts are doing is hard-coding a config override. Show Jim Bosch added a comment - Regarding your final comment, is it a question of ingestDecam.py vs ingest.py --config$OBS_DECAM/config/ingest.py ? I think we ask people what they prefer. for going with what's considered the best UI.  But also note that we can have different scripts for different cameras without having different tasks, if all the scripts are doing is hard-coding a config override.
Hide
John Parejko added a comment -

I am going to close this ticket as "Done": DM-23976 means that defects are being written in gen3 from obs_decam_data, and DM-22655 got us a mostly-working gen2to3 convert script for decam. DM-23616 includes notes about how to run the script for ap_verify_ci_hits2015, and took care of the groundwork of processing decam data with pipetask. The remaining known issues preventing us from knowing whether the conversion is correct and that we can run gen3 single frame processing are being dealt with on DM-23985, DM-23992, and DM-23983.

Show
John Parejko added a comment - I am going to close this ticket as "Done": DM-23976 means that defects are being written in gen3 from obs_decam_data, and DM-22655 got us a mostly-working gen2to3 convert script for decam. DM-23616 includes notes about how to run the script for ap_verify_ci_hits2015, and took care of the groundwork of processing decam data with pipetask. The remaining known issues preventing us from knowing whether the conversion is correct and that we can run gen3 single frame processing are being dealt with on DM-23985 , DM-23992 , and DM-23983 .

People

• Assignee:
John Parejko
Reporter:
Jim Bosch
Watchers:
Eric Bellm, Heather Kelly, Jim Bosch, John Parejko, John Swinbank, Krzysztof Findeisen, Meredith Rawls, Simon Krughoff, Tim Jenness
0 Vote for this issue
Watchers:
9 Start watching this issue

Dates

• Created:
Updated:
Resolved: