Details
-
Type:
Story
-
Status: Done
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: obs_decam
-
Labels:
Description
When processing data from the community pipeline instcal files, the ccdnum that the butler uses, and which is used to create a detector object, is not the same as the ccd number specified in the original image headers. For example,
>>> calexp = b.get("calexp", visit=406295, ccdnum=10)
|
>>> calexp.getDetector().getId()
|
10
|
>>> calexp.getDetector().getName()
|
'S22'
|
>>> calexp.getMetadata().toDict()['DETNAME']
|
'S22'
|
But if I check the headers, I get
>>> calexp.getMetadata().toDict()['CCDNUM']
|
11
|
>>> calexp.getMetadata().toDict()['DETECTOR']
|
'S3-164_107419-16-2'
|
where S3-164_107419-16-2 is the ID for the detector S23 aka ccdnum=11 (table of detector IDs).
The original source of this confusion seems to be in the decam policy entry for instcal, which specifies:
template: "%(visit)07d/instcal%(visit)07d.fits.fz[%(ccdnum)d]"
|
compared to the raw format, which is
template: "%(date)s/%(filter)s/decam%(visit)07d.fits.fz[%(hdu)d]"
|
It is likely necessary for instcal ingest to duplicate the behavior of the raw file ingest of MEF files, to prevent this conflation of HDU numbers with CCD Ids.
Attachments
Issue Links
- blocks
-
- Done
Activity
Looks good to me. Comments on the PRs.
Hsin-Fang Chiang
added a comment - Looks good to me. Comments on the PRs. Thanks for the review. Just as a heads-up, I realized after the review that I will also have to rebuild the registry in validation_data_decam.
Colin Slater
added a comment - Thanks for the review. Just as a heads-up, I realized after the review that I will also have to rebuild the registry in validation_data_decam. I take that as you are able to run validate_drp and other ci successfully 
Hsin-Fang Chiang
added a comment - I take that as you are able to run validate_drp and other ci successfully I updated the registry in validation_data_decam. In validate_drp, the script examples/runDecamTest.sh runs fine with the exception of one CCD that fails during processCcd (visit 176837 ccd 14), with the message:
INFO 2017-01-09T05:21:04.004 processCcd.calibrate.refObjLoader ({'ccdnum': 14, 'hdu': 14, 'visit': 176837})(loadReferenceObjects.py:213)- Loading reference objects using center (1022.5, 2046.5) pix = IcrsCoord(149.6572470, 2.6065910) sky and radius 0.171463569144 deg |
INFO 2017-01-09T05:21:04.004 processCcd.calibrate.refObjLoader ({'ccdnum': 14, 'hdu': 14, 'visit': 176837})(loadReferenceObjects.py:222)- Loaded 187 reference objects |
INFO 2017-01-09T05:21:04.004 processCcd.calibrate.astrometry.matcher ({'ccdnum': 14, 'hdu': 14, 'visit': 176837})(matchOptimisticB.py:288)- filterStars purged 0 reference stars, leaving 187 stars |
INFO 2017-01-09T05:21:05.005 processCcd.calibrate.astrometry.matcher ({'ccdnum': 14, 'hdu': 14, 'visit': 176837})(matchOptimisticB.py:302)- Purged 5071 unusable sources, leaving 279 usable sources |
FATAL 2017-01-09T05:21:09.009 processCcd ({'ccdnum': 14, 'hdu': 14, 'visit': 176837})(cmdLineTask.py:351)- Failed on dataId={'ccdnum': 14, 'hdu': 14, 'visit': 176837}: Unable to match sources |
Since this seems to also happen on master, I believe this is outside the scope of this ticket and I plan to merge as-is.
Michael Wood-Vasey, does this sound reasonable to you? I can't find the history of running decam with validate_drp, so I can't pin down when or where this error cropped up.
Colin Slater
added a comment - I updated the registry in validation_data_decam . In validate_drp , the script examples/runDecamTest.sh runs fine with the exception of one CCD that fails during processCcd (visit 176837 ccd 14), with the message:
INFO 2017 - 01 -09T05: 21 : 04.004 processCcd.calibrate.refObjLoader ({ 'ccdnum' : 14 , 'hdu' : 14 , 'visit' : 176837 })(loadReferenceObjects.py: 213 )- Loading reference objects using center ( 1022.5 , 2046.5 ) pix = IcrsCoord( 149.6572470 , 2.6065910 ) sky and radius 0.171463569144 deg
INFO 2017 - 01 -09T05: 21 : 04.004 processCcd.calibrate.refObjLoader ({ 'ccdnum' : 14 , 'hdu' : 14 , 'visit' : 176837 })(loadReferenceObjects.py: 222 )- Loaded 187 reference objects
INFO 2017 - 01 -09T05: 21 : 04.004 processCcd.calibrate.astrometry.matcher ({ 'ccdnum' : 14 , 'hdu' : 14 , 'visit' : 176837 })(matchOptimisticB.py: 288 )- filterStars purged 0 reference stars, leaving 187 stars
INFO 2017 - 01 -09T05: 21 : 05.005 processCcd.calibrate.astrometry.matcher ({ 'ccdnum' : 14 , 'hdu' : 14 , 'visit' : 176837 })(matchOptimisticB.py: 302 )- Purged 5071 unusable sources, leaving 279 usable sources
FATAL 2017 - 01 -09T05: 21 : 09.009 processCcd ({ 'ccdnum' : 14 , 'hdu' : 14 , 'visit' : 176837 })(cmdLineTask.py: 351 )- Failed on dataId={ 'ccdnum' : 14 , 'hdu' : 14 , 'visit' : 176837 }: Unable to match sources
Since this seems to also happen on master, I believe this is outside the scope of this ticket and I plan to merge as-is.
Michael Wood-Vasey , does this sound reasonable to you? I can't find the history of running decam with validate_drp, so I can't pin down when or where this error cropped up. Colin Slater Yes, this sounds reasonable.
Merge.
Michael Wood-Vasey
added a comment - Colin Slater Yes, this sounds reasonable.
Merge. Suggestions from review incorporated, tested on both instcal and raw files. Merged to master.
Colin Slater
added a comment - Suggestions from review incorporated, tested on both instcal and raw files. Merged to master. 
Hsin-Fang Chiang, would you have time to review this? I basically moved the code that saves the HDU number from the decam raw file ingest up into the base ingest script, so obs packages can either use or ignore the hdu number as they require.
The only change to testdata_decam is rebuilding the registry with the new ingest code. The existing test succeeded because it used ccd 1, where the hdu and ccdnum happened to match. I added a test for ccd 62.
I checked the community pipeline calib products, and they do seem to always have the HDU equal to the ccdnum, so I have not changed anything there.