# Make obs_decam handle raw data

XMLWordPrintable

#### Details

• Type: Story
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s:
• Labels:
None
• Story Points:
13
• Team:
Data Facility

#### Description

The current obs_decam expects instrument calibrated data from the community pipeline, i.e. it requires matching instcal (Instrument Calibrated), dqmask (the associated mask file), and wtmap (weight map) data from the same visit. This issue is to add functionality so that raw DECam images can be ingested into registry and retrieved by the data butler.

Practically, this will create new or expand existing sub-classes of CameraMapper and IngestTask.

A brief summary of changes:

• The unit test getRaw.py is updated and should pass, with DM-3196
• Working testdata_decam for the unit test is currently at lsst-dev /lsst8/testdata_decam and https://uofi.box.com/testdata-decam
• DecamInstcalMapper is renamed to DecamMapper, to reflect that Butler can also get "raw" now besides "instcal". Please update _mapper in your data repositories.
• To create a registry for raw data, run

  ingestImagesDecam.py /path/to/repo --mode=link --filetype="raw" /path/*.fits.fz 

• The default filetype is "instcal" for ingestImagesDecam.py, so previous use for instcal stays.

#### Activity

Hide
Simon Krughoff added a comment -

This looks good to me. I like having one ingester to ingest both types of data. I also think we should use Exposure instead of DecoratedImage, unless it won't work otherwise.

I didn't run the unit tests, but I can if you'd like me to for completeness.

Show
Simon Krughoff added a comment - This looks good to me. I like having one ingester to ingest both types of data. I also think we should use Exposure instead of DecoratedImage, unless it won't work otherwise. Some minor comments at: https://github.com/lsst/obs_decam/pull/2 I didn't run the unit tests, but I can if you'd like me to for completeness.
Hide
Hsin-Fang Chiang added a comment - - edited

Current CameraMapper standardize the input DecoratedImage into Exposure. So I guess when DecoratedImage is actually deprecated (DM-3190, DM-3191), something will be done with CameraMapper as well. DM-3191 seems to suggest there might be a new type to use.

Simply using Exposure in the policy file doesn't work. I will investigate more what else is needed to make it work (probably standardizing the raw imgaes in obs_decam).

Show
Hsin-Fang Chiang added a comment - - edited Thank you for your review! Current CameraMapper standardize the input DecoratedImage into Exposure. So I guess when DecoratedImage is actually deprecated ( DM-3190 , DM-3191 ), something will be done with CameraMapper as well. DM-3191 seems to suggest there might be a new type to use. Simply using Exposure in the policy file doesn't work. I will investigate more what else is needed to make it work (probably standardizing the raw imgaes in obs_decam).
Hide
Hsin-Fang Chiang added a comment -

By Paul Price,

DecoratedImage may be deprecated, but it's the only thing we have that fits here. The Exposure constructor will now produce a warning because they are not actually Exposures (e.g., no separate Wcs extension), and plain Image doesn't give us the FITS header.

I would leave it as DecoratedImage for now.

If it all looks good, I'll merge it to master.

Show
Hsin-Fang Chiang added a comment - By Paul Price , DecoratedImage may be deprecated, but it's the only thing we have that fits here. The Exposure constructor will now produce a warning because they are not actually Exposures (e.g., no separate Wcs extension), and plain Image doesn't give us the FITS header. I would leave it as DecoratedImage for now. If it all looks good, I'll merge it to master.
Hide
Simon Krughoff added a comment -

Sounds good. Merge away.

Show
Simon Krughoff added a comment - Sounds good. Merge away.
Hide
Hsin-Fang Chiang added a comment -

Merged to master.

  README.md | 6 +-  config/ingest.py | 1 +  decam/camGeom/camera.py | 2630 +++++++++++++++++-----------------  policy/DecamMapper.paf | 6 +-  python/lsst/obs/decam/decamMapper.py | 16 +-  python/lsst/obs/decam/ingest.py | 120 +-  tests/getRaw.py | 73 +-  7 files changed, 1435 insertions(+), 1417 deletions(-) 

Show
Hsin-Fang Chiang added a comment - Merged to master. README.md | 6 +- config/ingest.py | 1 + decam/camGeom/camera.py | 2630 +++++++++++++++++----------------- policy/DecamMapper.paf | 6 +- python/lsst/obs/decam/decamMapper.py | 16 +- python/lsst/obs/decam/ingest.py | 120 +- tests/getRaw.py | 73 +- 7 files changed, 1435 insertions(+), 1417 deletions(-)

#### People

Assignee:
Hsin-Fang Chiang
Reporter:
Hsin-Fang Chiang
Reviewers:
Simon Krughoff
Watchers:
David Nidever [X] (Inactive), Hsin-Fang Chiang, Simon Krughoff