XMLWordPrintable

## Details

• Type: Improvement
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s:
• Labels:
None
• Story Points:
8
• Sprint:
DRP S19-1, DRP S19-2
• Team:
Data Release Production

## Description

isrTask needs to be updated to work with a Gen3 butler as a valid pipelineTask.  DM-15862 will unify the ISR processing, so this should make all cameras functional with a Gen3 butler.

## Attachments

0.7 kB
188 kB
3. calibtool.py
9 kB
4. pickletool.py
0.6 kB

## Activity

Hide
Christopher Waters added a comment -

Working from the DM-15862 ticket branch, I've created a DM-16467 with an incomplete solution.  Currently, there are three issues:

1. raw data is loaded as a DecoratedImageU, but IsrTask expects it to be an Exposure.  Converting the received DecoratedImageU into an Exposure doesn't work, as the generated Exposure does not return a usable VisitInfo, and does not have a Detector associated.
2. Calibration data is not correctly returned by the query to the gen3 database.  A query/subquery of the form "select * from Exposure JOIN Dataset ON (valid_first < datetime_begin AND valid_last > datetime_end) WHERE Exposure.exposure = 903988 AND detector = 16 and dataset_type_name = 'bias';" needs to be done to get the correct (in this case) bias.  This will eventually supersede the IsrTask.readIsrData/IsrTask.getIsrExposure code, which currently fetches ~12 images/products.
3. Data queries that result in a single quanta to run (such as query two below, which restricts down to a single detector) fail to run as something in the code expects an array.  Removing the "Detector.detector=16" constraint allows it to continue to IsrTask.run().

Beyond these two issues, I can use the debugger to check that the code is reaching IsrTask.run(), and the graphs constructed for the raw input exposure look plausible based on my tests on ci_hsc.

Test commands for reference:

stac --clobber-config --clobber-versions -b ~/dev/DM-16467/ci_hsc/DATA/ -o test1 -i raw/HSC -p lsst.ip.isr --debug -L isr=debug -d "PhysicalFilter.physical_filter='HSC-I' AND Detector.detector=16 AND Exposure.exposure=903988" qgraph -t IsrTask --instrument-overrides --qgraph-dot ~/x.dot

stac --clobber-config --clobber-versions -b ~/dev/DM-16467/ci_hsc/DATA/ -o test1 -i raw/HSC -p lsst.ip.isr --debug -L isr=debug -d "PhysicalFilter.physical_filter='HSC-I' AND Detector.detector=16 AND Exposure.exposure=903988" run -t IsrTask --instrument-overrides

Show
Hide
Andy Salnikov added a comment -

Last point was caused by a bug in passing arguments to registry.associate() method. Fixed by DM-16481, already in pipe_supertask master, you can clone it if you prefer not to wait until next weekly release.

Show
Andy Salnikov added a comment - Last point was caused by a bug in passing arguments to registry.associate() method. Fixed by  DM-16481 , already in pipe_supertask master, you can clone it if you prefer not to wait until next weekly release.
Hide
Andy Salnikov added a comment -

For calibrations we need to add special handling of ExposureRange unit to pre-flight. I have made DM-16482 to work on that, it is of non-trivial kind but I hope to close it this week.

Show
Andy Salnikov added a comment - For calibrations we need to add special handling of ExposureRange unit to pre-flight. I have made  DM-16482 to work on that, it is of non-trivial kind but I hope to close it this week.
Hide
Andy Salnikov added a comment -

Christopher Waters, I have just merged daf_butler DM-16482 branch into master and that should take care of the calibration issue, flat/bias should now be correctly found and added to Quanta. Note that units for those should be:

 bias: Instrument, Detector, ExposureRange flat: Instrument, Detector, PhysicalFilter, ExposureRange 

Show
Andy Salnikov added a comment - Christopher Waters , I have just merged daf_butler DM-16482 branch into master and that should take care of the calibration issue, flat/bias should now be correctly found and added to Quanta. Note that units for those should be: bias: Instrument, Detector, ExposureRange flat: Instrument, Detector, PhysicalFilter, ExposureRange
Hide
Christopher Waters added a comment -

The complete ISR process does not work correctly at this time for this ticket branch:

• thumbnails do not have the needed daf_butler formatter to be written (DM-17381)
• fringe data for y-filter data is not read correctly (DM-17152)
• crosstalk sources are not generated for inter-CCD sources (DM-17169)

These issues do not prevent ci_hsc data from running, although additional calibration data must be ingested to add the brighter-fatter kernel and the HSC defects.  I have been adding those products using the attached python script, which will need to be superseded by a more permanent solution.  The commands to ingest products are:

 calibtool.py ./ci_hsc/DATA/gen3.sqlite3 pwd/ci_hsc/DATA/calib/bfKernel/brighter_fatter_kernel.pkl pwd/ci_hsc/DATA bfKernel --valid_first '1977-01-01T00:00:00.000001' --valid_last '2095-01-01T00:00:00.000001' --instrument HSC --init calibtool.py ./ci_hsc/DATA/gen3.sqlite3 pwd/ci_hsc/DATA/calib/defects/2013-01-31/defects_0.fits pwd/ci_hsc/DATA defects --valid_first '2013-01-31T00:00:00.000001' --valid_last '2014-04-02T23:59:59.999999' --instrument HSC --init --detector 0 calibtool.py ./ci_hsc/DATA/gen3.sqlite3 pwd/ci_hsc/DATA/calib/defects/2013-01-31/defects_1.fits pwd/ci_hsc/DATA defects --valid_first '2013-01-31T00:00:00.000001' --valid_last '2014-04-02T23:59:59.999999' --instrument HSC --detector 1 

The gen2 behavior has not changed, and a comparison of output files shows that both generations have pixel-identical results, with the exception of changes in mask plane labeling.

Show
Christopher Waters added a comment - The complete ISR process does not work correctly at this time for this ticket branch: thumbnails do not have the needed daf_butler formatter to be written ( DM-17381 ) fringe data for y-filter data is not read correctly ( DM-17152 ) crosstalk sources are not generated for inter-CCD sources ( DM-17169 ) These issues do not prevent ci_hsc data from running, although additional calibration data must be ingested to add the brighter-fatter kernel and the HSC defects.  I have been adding those products using the attached python script, which will need to be superseded by a more permanent solution.  The commands to ingest products are:     calibtool.py ./ci_hsc/DATA/gen3.sqlite3 pwd/ci_hsc/DATA/calib/bfKernel/brighter_fatter_kernel.pkl pwd/ci_hsc/DATA bfKernel --valid_first '1977-01-01T00:00:00.000001' --valid_last '2095-01-01T00:00:00.000001' --instrument HSC --init calibtool.py ./ci_hsc/DATA/gen3.sqlite3 pwd/ci_hsc/DATA/calib/defects/ 2013 - 01 - 31 /defects_0.fits pwd/ci_hsc/DATA defects --valid_first '2013-01-31T00:00:00.000001' --valid_last '2014-04-02T23:59:59.999999' --instrument HSC --init --detector 0 calibtool.py ./ci_hsc/DATA/gen3.sqlite3 pwd/ci_hsc/DATA/calib/defects/ 2013 - 01 - 31 /defects_1.fits pwd/ci_hsc/DATA defects --valid_first '2013-01-31T00:00:00.000001' --valid_last '2014-04-02T23:59:59.999999' --instrument HSC --detector 1 The gen2 behavior has not changed, and a comparison of output files shows that both generations have pixel-identical results, with the exception of changes in mask plane labeling.
Hide
Simon Krughoff added a comment - - edited

Looks pretty good to me. I have put some comments in the PR. They are mostly suggestion level.

The most concerning thing to me is the hard coding of obs_subaru information, but, as is mentioned on this ticket, we are still shimming some aspect of things.

Show
Simon Krughoff added a comment - - edited Looks pretty good to me. I have put some comments in the PR. They are mostly suggestion level. The most concerning thing to me is the hard coding of obs_subaru information, but, as is mentioned on this ticket, we are still shimming some aspect of things.
Hide
Christopher Waters added a comment - - edited

To be clear:  while testing ISR on ci_hsc, I often needed to start over from scratch and try again.  To make this easier and keep things consistent, I migrated as much as possible into script files.  This meant I could regenerate a usable ci_hsc gen3 repository for ISR by running:

 #Clear old repo: rm ./ci_hsc/DATA/gen3.sqlite3 ./ci_hsc/DATA/butler.yaml #Regenerate repo: cd ./ci_hsc && makeButlerRepo.py DATA && python ./bin/gen3.py #Ingest calibs: bash ./add_bfKernel.sh bash ./add_defects.sh 

The two bash scripts encapsulate the calibtool commands for both calib types, and converts the brighter-fatter kernel to a python3 pickle using pickletool.

Show
Christopher Waters added a comment - - edited To be clear:  while testing ISR on ci_hsc, I often needed to start over from scratch and try again.  To make this easier and keep things consistent, I migrated as much as possible into script files.  This meant I could regenerate a usable ci_hsc gen3 repository for ISR by running:   #Clear old repo: rm ./ci_hsc/DATA/gen3.sqlite3 ./ci_hsc/DATA/butler.yaml #Regenerate repo: cd ./ci_hsc && makeButlerRepo.py DATA && python ./bin/gen3.py #Ingest calibs: bash ./add_bfKernel.sh bash ./add_defects.sh  The two bash scripts encapsulate the calibtool commands for both calib types, and converts the brighter-fatter kernel to a python3 pickle using pickletool.

## People

• Assignee:
Christopher Waters
Reporter:
Christopher Waters
Reviewers:
Simon Krughoff
Watchers:
Andy Salnikov, Christopher Waters, Simon Krughoff