# obs_decam assigns wrong ccdnum to instcal data

XMLWordPrintable

#### Details

• Type: Story
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s:
• Labels:

#### Description

When processing data from the community pipeline instcal files, the ccdnum that the butler uses, and which is used to create a detector object, is not the same as the ccd number specified in the original image headers. For example,

 >>> calexp = b.get("calexp", visit=406295, ccdnum=10) >>> calexp.getDetector().getId() 10 >>> calexp.getDetector().getName() 'S22' >>> calexp.getMetadata().toDict()['DETNAME'] 'S22' 

But if I check the headers, I get

 >>> calexp.getMetadata().toDict()['CCDNUM'] 11 >>> calexp.getMetadata().toDict()['DETECTOR'] 'S3-164_107419-16-2' 

where S3-164_107419-16-2 is the ID for the detector S23 aka ccdnum=11 (table of detector IDs).

The original source of this confusion seems to be in the decam policy entry for instcal, which specifies:

 template: "%(visit)07d/instcal%(visit)07d.fits.fz[%(ccdnum)d]" 

compared to the raw format, which is

 template: "%(date)s/%(filter)s/decam%(visit)07d.fits.fz[%(hdu)d]" 

It is likely necessary for instcal ingest to duplicate the behavior of the raw file ingest of MEF files, to prevent this conflation of HDU numbers with CCD Ids.

#### Activity

Hide
Colin Slater added a comment -

Hsin-Fang Chiang, would you have time to review this? I basically moved the code that saves the HDU number from the decam raw file ingest up into the base ingest script, so obs packages can either use or ignore the hdu number as they require.

The only change to testdata_decam is rebuilding the registry with the new ingest code. The existing test succeeded because it used ccd 1, where the hdu and ccdnum happened to match. I added a test for ccd 62.

I checked the community pipeline calib products, and they do seem to always have the HDU equal to the ccdnum, so I have not changed anything there.

Show
Colin Slater added a comment - Hsin-Fang Chiang , would you have time to review this? I basically moved the code that saves the HDU number from the decam raw file ingest up into the base ingest script, so obs packages can either use or ignore the hdu number as they require. The only change to testdata_decam is rebuilding the registry with the new ingest code. The existing test succeeded because it used ccd 1, where the hdu and ccdnum happened to match. I added a test for ccd 62. I checked the community pipeline calib products, and they do seem to always have the HDU equal to the ccdnum, so I have not changed anything there.
Hide
Hsin-Fang Chiang added a comment -

Looks good to me. Comments on the PRs.

Show
Hsin-Fang Chiang added a comment - Looks good to me. Comments on the PRs.
Hide
Colin Slater added a comment -

Thanks for the review. Just as a heads-up, I realized after the review that I will also have to rebuild the registry in validation_data_decam.

Show
Colin Slater added a comment - Thanks for the review. Just as a heads-up, I realized after the review that I will also have to rebuild the registry in validation_data_decam.
Hide
Hsin-Fang Chiang added a comment -

I take that as you are able to run validate_drp and other ci successfully

Show
Hsin-Fang Chiang added a comment - I take that as you are able to run validate_drp and other ci successfully
Hide
Colin Slater added a comment -

I updated the registry in validation_data_decam. In validate_drp, the script examples/runDecamTest.sh runs fine with the exception of one CCD that fails during processCcd (visit 176837 ccd 14), with the message:

 INFO 2017-01-09T05:21:04.004 processCcd.calibrate.refObjLoader ({'ccdnum': 14, 'hdu': 14, 'visit': 176837})(loadReferenceObjects.py:213)- Loading reference objects using center (1022.5, 2046.5) pix = IcrsCoord(149.6572470, 2.6065910) sky and radius 0.171463569144 deg INFO 2017-01-09T05:21:04.004 processCcd.calibrate.refObjLoader ({'ccdnum': 14, 'hdu': 14, 'visit': 176837})(loadReferenceObjects.py:222)- Loaded 187 reference objects INFO 2017-01-09T05:21:04.004 processCcd.calibrate.astrometry.matcher ({'ccdnum': 14, 'hdu': 14, 'visit': 176837})(matchOptimisticB.py:288)- filterStars purged 0 reference stars, leaving 187 stars INFO 2017-01-09T05:21:05.005 processCcd.calibrate.astrometry.matcher ({'ccdnum': 14, 'hdu': 14, 'visit': 176837})(matchOptimisticB.py:302)- Purged 5071 unusable sources, leaving 279 usable sources FATAL 2017-01-09T05:21:09.009 processCcd ({'ccdnum': 14, 'hdu': 14, 'visit': 176837})(cmdLineTask.py:351)- Failed on dataId={'ccdnum': 14, 'hdu': 14, 'visit': 176837}: Unable to match sources 

Since this seems to also happen on master, I believe this is outside the scope of this ticket and I plan to merge as-is.
Michael Wood-Vasey, does this sound reasonable to you? I can't find the history of running decam with validate_drp, so I can't pin down when or where this error cropped up.

Show
Colin Slater added a comment - I updated the registry in validation_data_decam . In validate_drp , the script examples/runDecamTest.sh runs fine with the exception of one CCD that fails during processCcd (visit 176837 ccd 14), with the message: INFO 2017 - 01 -09T05: 21 : 04.004 processCcd.calibrate.refObjLoader ({ 'ccdnum' : 14 , 'hdu' : 14 , 'visit' : 176837 })(loadReferenceObjects.py: 213 )- Loading reference objects using center ( 1022.5 , 2046.5 ) pix = IcrsCoord( 149.6572470 , 2.6065910 ) sky and radius 0.171463569144 deg INFO 2017 - 01 -09T05: 21 : 04.004 processCcd.calibrate.refObjLoader ({ 'ccdnum' : 14 , 'hdu' : 14 , 'visit' : 176837 })(loadReferenceObjects.py: 222 )- Loaded 187 reference objects INFO 2017 - 01 -09T05: 21 : 04.004 processCcd.calibrate.astrometry.matcher ({ 'ccdnum' : 14 , 'hdu' : 14 , 'visit' : 176837 })(matchOptimisticB.py: 288 )- filterStars purged 0 reference stars, leaving 187 stars INFO 2017 - 01 -09T05: 21 : 05.005 processCcd.calibrate.astrometry.matcher ({ 'ccdnum' : 14 , 'hdu' : 14 , 'visit' : 176837 })(matchOptimisticB.py: 302 )- Purged 5071 unusable sources, leaving 279 usable sources FATAL 2017 - 01 -09T05: 21 : 09.009 processCcd ({ 'ccdnum' : 14 , 'hdu' : 14 , 'visit' : 176837 })(cmdLineTask.py: 351 )- Failed on dataId={ 'ccdnum' : 14 , 'hdu' : 14 , 'visit' : 176837 }: Unable to match sources Since this seems to also happen on master, I believe this is outside the scope of this ticket and I plan to merge as-is. Michael Wood-Vasey , does this sound reasonable to you? I can't find the history of running decam with validate_drp, so I can't pin down when or where this error cropped up.
Hide
Michael Wood-Vasey added a comment -

Colin Slater Yes, this sounds reasonable.
Merge.

Show
Michael Wood-Vasey added a comment - Colin Slater Yes, this sounds reasonable. Merge.
Hide
Colin Slater added a comment -

Suggestions from review incorporated, tested on both instcal and raw files. Merged to master.

Show
Colin Slater added a comment - Suggestions from review incorporated, tested on both instcal and raw files. Merged to master.

#### People

Assignee:
Colin Slater
Reporter:
Colin Slater
Reviewers:
Hsin-Fang Chiang
Watchers:
Colin Slater, Hsin-Fang Chiang, Michael Wood-Vasey