# Initial code change to run ISR with DECam raw data

XMLWordPrintable

#### Details

• Type: Story
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s:
• Labels:
None
• Story Points:
14
• Team:
Data Facility

#### Description

This ticket is for implementing changes in obs_decam in order to run processCcd with raw DECam data.

Changes are mostly in DecamMapper and a new class DecamIsrTask is added. A test to retrieve defects with Butler is also added. (testdata_decam is at lsst-dev /lsst8/testdata_decam/ )

The pixels with bit 1 (bad, hot/dead pixel/column) from the Community Pipeline Bad Pixel Masks are used as bad pixels. The CP BPM fits files are directly used as defect files. Due to their large size, they probably should not go into obs_decam repository so they are treated similarly as other calibration products.

With the changes of this ticket, the following ingest defects files into calibRegistry:

 ingestCalibs.py . --calibType defect path-to-bpm/*fits 

and the following should run past ISR:

 processCcd.py . --config isr.doFringe=False isr.assembleCcd.setGain=False calibrate.doPhotoCal=False calibrate.doAstrometry=False calibrate.measurePsf.starSelector.name="secondMoment" 

Running fringe correction with DECam raw data will be in future tickets (DM-4223 and possibly more). Also this ticket does not cover implementing or porting new ISR functionalities that haven't yet been included in ip_isr (such as crosstalk).

#### Activity

Hide
Russell Owen added a comment -

This looks great. Thanks for the improvements. I have a few minor things to ask about, but nothing that would stop you from merging now if you like.

You say "For defects, the products are valid from the date they are registered until the next date. For other calibration products, a midpoint is used to fix the conflicts.". On the whole this is pretty clear, but I suggest trying to use a different phrase than "fix the conflicts". Do we actually expect conflicts? I thought data products had a start date, but not an end date, so conflicts will not occur. Perhaps you could say "For most calibration products, the calibration data with whose start date is nearest the date of the observation is used. However, defects files are valid from their start date until they are superseded by subsequent defect data".

The --calibType argument help is much clearer, but I still suggest explicitly saying that the OBSTYPE header entry is used (as per my original review).

### obs_decam

Would it make sense to shorten the key "ccdnum" down to simply "ccd"? (Certainly not if you already use the key "ccd" for something else).

Why in std_raw do you copy over all metadata? I'm referring to the following snippet:

  for kw in md0.paramNames():  if kw not in md.paramNames():  md.add(kw, md0.get(kw)) 

isr.py: I am a bit surprised that you require that the amount bias and flats are trimmed is the same in x and y. After all, you had separate parameters for those earlier. But if you are positive this is a safe assumption then I have no objection. If you do want to allow them to be different the code would be easy to expand to return a tuple of x, y trim amounts.

Show
Russell Owen added a comment - This looks great. Thanks for the improvements. I have a few minor things to ask about, but nothing that would stop you from merging now if you like. pipe_tasks You say "For defects, the products are valid from the date they are registered until the next date. For other calibration products, a midpoint is used to fix the conflicts.". On the whole this is pretty clear, but I suggest trying to use a different phrase than "fix the conflicts". Do we actually expect conflicts? I thought data products had a start date, but not an end date, so conflicts will not occur. Perhaps you could say "For most calibration products, the calibration data with whose start date is nearest the date of the observation is used. However, defects files are valid from their start date until they are superseded by subsequent defect data". The --calibType argument help is much clearer, but I still suggest explicitly saying that the OBSTYPE header entry is used (as per my original review). obs_decam Would it make sense to shorten the key "ccdnum" down to simply "ccd"? (Certainly not if you already use the key "ccd" for something else). Why in std_raw do you copy over all metadata? I'm referring to the following snippet: for kw in md0.paramNames(): if kw not in md.paramNames(): md.add(kw, md0.get(kw)) isr.py: I am a bit surprised that you require that the amount bias and flats are trimmed is the same in x and y. After all, you had separate parameters for those earlier. But if you are positive this is a safe assumption then I have no objection. If you do want to allow them to be different the code would be easy to expand to return a tuple of x, y trim amounts.
Hide
Russell Owen added a comment - - edited

One more thing that Simon Krughoff pointed out: in the paf file you use "[%(hdu)d]" for raw images, but "[%(ccdnum)d]" for all other image types. It seems as if the keys "hdu" and "ccdnum" are being used to mean the same thing. I assume this is intentional, but it seems odd and I just wanted to be sure

Show
Russell Owen added a comment - - edited One more thing that Simon Krughoff pointed out: in the paf file you use " [%(hdu)d] " for raw images, but " [%(ccdnum)d] " for all other image types. It seems as if the keys "hdu" and "ccdnum" are being used to mean the same thing. I assume this is intentional, but it seems odd and I just wanted to be sure
Hide
Hsin-Fang Chiang added a comment - - edited

Thanks a lot for your speedy reply! Sorry I didn't mean to rush you.

I see what you mean the wording about "fixing the conflicts" is not clear, I will edit that.

As far as I know at this moment pipe_tasks/ingestCalibs is only used by obs_decam. I was thinking if other obs_* packages want to use it in the future, they will likely have to override CalibsPasreTask.getCalibType and they might want to use a different header keyword or they can also use filenames. That's why I didn't want to say "OBSTYPE" header explicitly.

Would it make sense to shorten the key "ccdnum" down to simply "ccd"? (Certainly not if you already use the key "ccd" for something else).

The short answer is "ccd" is used for something else.
A bit longer answer is that there was some confusing history about the use of "ccdnum" and "ccd" in obs_decam.
With my limited knowledge I think that use of "ccd" can be eliminated altogether and then replace ccdnum by ccd (I think I know how and sometimes I really hope to do so). However I'm not completely sure about all the consequenses. Also I don't want to add more confusions.

Why in std_raw do you copy over all metadata?

There are more header keywords existing only in the zeroth header but not in others. So far I only encountered problems due to EXPTIME and MJD-OBS. I was thinking it doesn't hurt to copy all non-repeating keywords. (a bit waste of space...)

isr.py: I am a bit surprised that you require that the amount bias and flats are trimmed is the same in x and y. After all, you had separate parameters for those earlier. But if you are positive this is a safe assumption then I have no objection. If you do want to allow them to be different the code would be easy to expand to return a tuple of x, y trim amounts.

True, I guess I preferred to have a strict assumption (and force people to look at the code again when Community Pipeline changes their settings )

in the paf file you use "[%(hdu)d]" for raw images, but "[%(ccdnum)d]" for all other image types.

The keys "hdu" and "ccdnum" have different meanings. "[%(ccdnum)d]" in the instcal/dqmask/wtmap template is a known bug. "[%(ccdnum)d]" in the bias/flat template is correct (so far as I understand) and was my convenient choice. If we want to be proactive to future Community Pipeline changes, we may want to change all of them to "hdu" (changes in ingest task needed).

Show
Hide
Hsin-Fang Chiang added a comment -

use a different phrase than "fix the conflicts". Do we actually expect conflicts?

I guess the confusion might have also come from the method name "resolveOverlaps". Previously it was indeed for fixing overlaps in validity ranges. The validity range is set through the file header or filename information or/and specified through user's input (validity period in days), so the ranges can indeed overlap. For example if there are daily calibration products but the user input a validity period of 10 days.

But the wording was not accurate for defects products. I renamed the method name and edited the comments. Hopefully it's more clear now.

A bit more on hdu and ccdnum...

"hdu" refers to the HDU number in the FITS files. They are not necessarily ordered by ccdnum, which corresponds to a specific and fixed CCD of DECam.
In Community Pipeline bias and flat data I've checked so far, they are ordered nicely and always the same (hdu == ccdnum). Raw data have the order shuffled.

Show
Hsin-Fang Chiang added a comment - use a different phrase than "fix the conflicts". Do we actually expect conflicts? I guess the confusion might have also come from the method name "resolveOverlaps". Previously it was indeed for fixing overlaps in validity ranges. The validity range is set through the file header or filename information or/and specified through user's input (validity period in days), so the ranges can indeed overlap. For example if there are daily calibration products but the user input a validity period of 10 days. But the wording was not accurate for defects products. I renamed the method name and edited the comments. Hopefully it's more clear now. A bit more on hdu and ccdnum... "hdu" refers to the HDU number in the FITS files. They are not necessarily ordered by ccdnum, which corresponds to a specific and fixed CCD of DECam. In Community Pipeline bias and flat data I've checked so far, they are ordered nicely and always the same (hdu == ccdnum). Raw data have the order shuffled.
Hide
Hsin-Fang Chiang added a comment -

Thanks a lot for your careful review and thoughtful comments! Changes are merged to master.

  python/lsst/pipe/tasks/ingestCalibs.py | 47 ++++++++++++++++++++++++----------  1 file changed, 34 insertions(+), 13 deletions(-) 

### obs_decam

 README.md | 2 +-  config/ingestCalibs.py | 2 +-  config/processCcd.py | 11 ++  decam/camGeom/N1.fits | 2 +-  .... (in total 62 fits files of the updated CCD files)  decam/segmentfile.txt | 248 +++++++++++++++++-----------------  policy/DecamMapper.paf | 36 ++++-  python/lsst/obs/decam/decamMapper.py | 83 ++++++++++--  python/lsst/obs/decam/ingest.py | 22 ++-  python/lsst/obs/decam/ingestCalibs.py | 23 ++++  python/lsst/obs/decam/isr.py | 80 ++++++++++-  tests/getRaw.py | 7 +  72 files changed, 440 insertions(+), 214 deletions(-) 

Show
Hsin-Fang Chiang added a comment - Thanks a lot for your careful review and thoughtful comments! Changes are merged to master. pipe_tasks python/lsst/pipe/tasks/ingestCalibs.py | 47 ++++++++++++++++++++++++---------- 1 file changed, 34 insertions(+), 13 deletions(-) obs_decam README.md | 2 +- config/ingestCalibs.py | 2 +- config/processCcd.py | 11 ++ decam/camGeom/N1.fits | 2 +- .... (in total 62 fits files of the updated CCD files) decam/segmentfile.txt | 248 +++++++++++++++++----------------- policy/DecamMapper.paf | 36 ++++- python/lsst/obs/decam/decamMapper.py | 83 ++++++++++-- python/lsst/obs/decam/ingest.py | 22 ++- python/lsst/obs/decam/ingestCalibs.py | 23 ++++ python/lsst/obs/decam/isr.py | 80 ++++++++++- tests/getRaw.py | 7 + 72 files changed, 440 insertions(+), 214 deletions(-)

#### People

Assignee:
Hsin-Fang Chiang
Reporter:
Hsin-Fang Chiang
Reviewers:
Russell Owen
Watchers:
Hsin-Fang Chiang, Paul Price, Russell Owen