# Bug fix and improvement for DECam processing

XMLWordPrintable

#### Details

• Type: Story
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s:
• Labels:
None
• Story Points:
6
• Team:
Data Facility

#### Description

• Bug fix in DecamMapper policy of fringe dataset
• Bug fix on translating Community Pipeline's Bad Pixel Mask (BPM) — Previously in DM-4191 I looked up the wrong table for the BPM bit definition.
• Flag the potentially problematic edge pixels as SUSPECT (DM-4515)

#### Attachments

1. numEdgeSuspect30.png
215 kB
2. numEdgeSuspect35.png
216 kB

#### Activity

Hide
Hsin-Fang Chiang added a comment -

Simon Krughoff would you mind reviewing these changes in branch tickets/DM-4559? These are
a bunch of miscellaneous small fixes, mostly for ISR processing, with the exception of the last one for making coadd.

In response to the previous conversations about defects format, I added a script to output the defects as in the format of obs_subaru/hsc. There can be redundant files in the output files. I think only the diff is recorded for HSC, while my script outputs everything at each date, but the resultant defects should be the same.

I decided not to switch the defect format in obs_decam for now when we still rely on the Community Pipeline calibration files, but am happy to see it change in the future.
For now I would rather make the users responbile of maintaining their defects files in their data repositories, just like other calibration files (flat/bias/fringe). The script may be a first step for switching to that format.

Show
Hsin-Fang Chiang added a comment - Simon Krughoff would you mind reviewing these changes in branch tickets/ DM-4559 ? These are a bunch of miscellaneous small fixes, mostly for ISR processing, with the exception of the last one for making coadd. In response to the previous conversations about defects format, I added a script to output the defects as in the format of obs_subaru/hsc. There can be redundant files in the output files. I think only the diff is recorded for HSC, while my script outputs everything at each date, but the resultant defects should be the same. I decided not to switch the defect format in obs_decam for now when we still rely on the Community Pipeline calibration files, but am happy to see it change in the future. For now I would rather make the users responbile of maintaining their defects files in their data repositories, just like other calibration files (flat/bias/fringe). The script may be a first step for switching to that format.
Hide
Colin Slater added a comment -

I just caught an issue in the decam policy file, since it's only two lines Hsin-Fang Chiang maybe you could fold this into your updates? The issue is that deepDiff_config has python: "lsst.pex.config.Config", but when I try to run image differencing it complains and it should be "lsst.pipe.tasks.imageDifference.ImageDifferenceConfig". Changing the policy file takes care of the issue when I test it.

Looking ahead, I see that the same issue exists with goodSeeingDiff_config; I think it should get the same fix.

Show
Colin Slater added a comment - I just caught an issue in the decam policy file, since it's only two lines Hsin-Fang Chiang maybe you could fold this into your updates? The issue is that deepDiff_config has python: "lsst.pex.config.Config" , but when I try to run image differencing it complains and it should be "lsst.pipe.tasks.imageDifference.ImageDifferenceConfig" . Changing the policy file takes care of the issue when I test it. Looking ahead, I see that the same issue exists with goodSeeingDiff_config ; I think it should get the same fix.
Hide
Hsin-Fang Chiang added a comment - - edited

Colin Slater reported errors from here without the config type fix:

 assert type(config)==lsst.pipe.tasks.imageDifference.ImageDifferenceConfig, 'config is of type %s.%s instead of lsst.pipe.tasks.imageDifference.ImageDifferenceC onfig' % (type(config).__module__, type(config).__name__) 

Show
Hsin-Fang Chiang added a comment - - edited Colin Slater reported errors from here without the config type fix: assert type(config)==lsst.pipe.tasks.imageDifference.ImageDifferenceConfig, 'config is of type %s.%s instead of lsst.pipe.tasks.imageDifference.ImageDifferenceC onfig' % (type(config).__module__, type(config).__name__)
Hide
Hsin-Fang Chiang added a comment -

I added the output files of examples/genDefectText.py script run with my edited Community Pipeline BPM files. They are in the last commit of branch u/hfc/DM-4559-example just to show how the output files look like and are not meant to be merged to obs_decam.

Show
Hsin-Fang Chiang added a comment - I added the output files of examples/genDefectText.py script run with my edited Community Pipeline BPM files. They are in the last commit of branch u/hfc/ DM-4559 -example just to show how the output files look like and are not meant to be merged to obs_decam .
Hide
Hsin-Fang Chiang added a comment -

• If I understand correctly, DM-4180 and subsequent refactoring will change this.
• The methods bypass_deepCoaddId, bypass_deepCoaddId_bits, and _computeCoaddExposureId are implemented similarly in obs_subaru, obs_sdss, obs_cfht, and obs_lsstSim. If https://dev.lsstcorp.org/trac/ticket/2797 or something similar is not happening, why not add them to CameraMapper?
Show
Hsin-Fang Chiang added a comment - About adding coadd data products: If I understand correctly, DM-4180 and subsequent refactoring will change this. The methods bypass_deepCoaddId , bypass_deepCoaddId_bits , and _computeCoaddExposureId are implemented similarly in obs_subaru , obs_sdss , obs_cfht , and obs_lsstSim . If https://dev.lsstcorp.org/trac/ticket/2797 or something similar is not happening, why not add them to CameraMapper ?
Hide
Simon Krughoff added a comment -

I have a couple of comments in the pull request, but otherwise looks fine. I also had the question about whether the bypass methods should also be moved to CameraMapper. Feel free to merge after you address my comments.

Show
Simon Krughoff added a comment - I have a couple of comments in the pull request, but otherwise looks fine. I also had the question about whether the bypass methods should also be moved to CameraMapper. Feel free to merge after you address my comments.
Hide
Hsin-Fang Chiang added a comment -

One possible reason the bypass methods are not be added to CameraMapper is that the mapper policy file lives in each obs package and is camera-specific. It can't work alone without adding those datasets to the policy file. I could understand adding them to CameraMapper would look somewhat odd, although having some default implementations in CameraMapper would be convenient for new cameras.

Probably DM-4180 will change that. I guess we can revisit this after DM-4180.
(if it's better to create a ticket now, I can do that too)

Show
Hsin-Fang Chiang added a comment - Thank you for your review! One possible reason the bypass methods are not be added to CameraMapper is that the mapper policy file lives in each obs package and is camera-specific. It can't work alone without adding those datasets to the policy file. I could understand adding them to CameraMapper would look somewhat odd, although having some default implementations in CameraMapper would be convenient for new cameras. Probably DM-4180 will change that. I guess we can revisit this after DM-4180 . (if it's better to create a ticket now, I can do that too)
Hide
Simon Krughoff added a comment -

I'd say don't worry about a new ticket. I think, since we've added these to most obs packages by now anyway, we can just see how the tickets you mention work out.

Show
Simon Krughoff added a comment - I'd say don't worry about a new ticket. I think, since we've added these to most obs packages by now anyway, we can just see how the tickets you mention work out.
Hide
Hsin-Fang Chiang added a comment -

Simon Krughoff I wasn't sure if I understand your note on testing the edge bit setting. Do you mean testing if meas.algorithms.detection.SourceDetectionTask.setEdgeBits correctly does what it says? If so, I am thinking maybe that test would suit better in meas_algorithms?

Show
Hsin-Fang Chiang added a comment - Simon Krughoff I wasn't sure if I understand your note on testing the edge bit setting. Do you mean testing if meas.algorithms.detection.SourceDetectionTask.setEdgeBits correctly does what it says? If so, I am thinking maybe that test would suit better in meas_algorithms?
Hide
Hsin-Fang Chiang added a comment -

oops my force-push seemed to have eliminated your notes...

https://github.com/lsst/obs_decam/pull/13/files#diff-c8df75c8df81724a84a227b16263525bR145

Show
Hide
Hsin-Fang Chiang added a comment -

I added one line comment hopefully to make it more clear, so the previous link is now off by one line, but you get the idea.

Show
Hsin-Fang Chiang added a comment - I added one line comment hopefully to make it more clear, so the previous link is now off by one line, but you get the idea.
Hide
Hsin-Fang Chiang added a comment -

I merged the changes to master.

obs_decam

  README.md | 10 +-  examples/genDefectText.py | 49 ++++++++  policy/DecamMapper.paf | 225 ++++++++++++++++++++++++++++++++++-  python/lsst/obs/decam/decamMapper.py | 48 +++++++-  python/lsst/obs/decam/isr.py | 26 ++++  tests/getRaw.py | 2 +-  6 files changed, 351 insertions(+), 9 deletions(-) 

testdata_decam

  calib/calibRegistry.sqlite3 | 2 +-  calib/fringe/DECam_Master_20131115v1-zG_ci_z_01.fits | 3 +++  calib/fringe/DECam_Master_20131115v1-zG_ci_z_13.fits | 3 ---  3 files changed, 4 insertions(+), 4 deletions(-) 

I didn't add a test in meas.algorithms.detection.SourceDetectionTask.setEdgeBits. A new ticket may be filed. As for its use in obs.decam.isr, I added some comments in the code to describe what is going on there. Images were also visually inspected to check it does do what we intend. For example the attacked screenshots and were created with isr.numEdgeSuspect=30 and isr.numEdgeSuspect=35, respectively. Yellow is SUSPECT, red (which looks more like orange with my transparency setting) is BAD, and green is INTRP. The yellow mask covers 30 and 30 pixels wide as expected.

Show
Hsin-Fang Chiang added a comment - I merged the changes to master. obs_decam README.md | 10 +- examples/genDefectText.py | 49 ++++++++ policy/DecamMapper.paf | 225 ++++++++++++++++++++++++++++++++++- python/lsst/obs/decam/decamMapper.py | 48 +++++++- python/lsst/obs/decam/isr.py | 26 ++++ tests/getRaw.py | 2 +- 6 files changed, 351 insertions(+), 9 deletions(-) testdata_decam calib/calibRegistry.sqlite3 | 2 +- calib/fringe/DECam_Master_20131115v1-zG_ci_z_01.fits | 3 +++ calib/fringe/DECam_Master_20131115v1-zG_ci_z_13.fits | 3 --- 3 files changed, 4 insertions(+), 4 deletions(-) I didn't add a test in meas.algorithms.detection.SourceDetectionTask.setEdgeBits . A new ticket may be filed. As for its use in obs.decam.isr , I added some comments in the code to describe what is going on there. Images were also visually inspected to check it does do what we intend. For example the attacked screenshots and were created with isr.numEdgeSuspect=30 and isr.numEdgeSuspect=35 , respectively. Yellow is SUSPECT , red (which looks more like orange with my transparency setting) is BAD , and green is INTRP . The yellow mask covers 30 and 30 pixels wide as expected.

#### People

Assignee:
Hsin-Fang Chiang
Reporter:
Hsin-Fang Chiang
Reviewers:
Simon Krughoff
Watchers:
Colin Slater, Hsin-Fang Chiang, Simon Krughoff