Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-3462

Make obs_decam handle raw data

    XMLWordPrintable

Details

    • Story
    • Status: Done
    • Resolution: Done
    • None
    • obs_decam
    • None

    Description

      The current obs_decam expects instrument calibrated data from the community pipeline, i.e. it requires matching instcal (Instrument Calibrated), dqmask (the associated mask file), and wtmap (weight map) data from the same visit. This issue is to add functionality so that raw DECam images can be ingested into registry and retrieved by the data butler.

      Practically, this will create new or expand existing sub-classes of CameraMapper and IngestTask.

      A brief summary of changes:

      • The unit test getRaw.py is updated and should pass, with DM-3196
      • Working testdata_decam for the unit test is currently at lsst-dev /lsst8/testdata_decam and https://uofi.box.com/testdata-decam
      • DecamInstcalMapper is renamed to DecamMapper, to reflect that Butler can also get "raw" now besides "instcal". Please update _mapper in your data repositories.
      • To create a registry for raw data, run

           ingestImagesDecam.py /path/to/repo --mode=link --filetype="raw" /path/*.fits.fz
        

      • The default filetype is "instcal" for ingestImagesDecam.py, so previous use for instcal stays.

      Attachments

        Issue Links

          Activity

            This looks good to me. I like having one ingester to ingest both types of data. I also think we should use Exposure instead of DecoratedImage, unless it won't work otherwise.

            Some minor comments at: https://github.com/lsst/obs_decam/pull/2

            I didn't run the unit tests, but I can if you'd like me to for completeness.

            krughoff Simon Krughoff (Inactive) added a comment - This looks good to me. I like having one ingester to ingest both types of data. I also think we should use Exposure instead of DecoratedImage, unless it won't work otherwise. Some minor comments at: https://github.com/lsst/obs_decam/pull/2 I didn't run the unit tests, but I can if you'd like me to for completeness.
            hchiang2 Hsin-Fang Chiang added a comment - - edited

            Thank you for your review!

            Current CameraMapper standardize the input DecoratedImage into Exposure. So I guess when DecoratedImage is actually deprecated (DM-3190, DM-3191), something will be done with CameraMapper as well. DM-3191 seems to suggest there might be a new type to use.

            Simply using Exposure in the policy file doesn't work. I will investigate more what else is needed to make it work (probably standardizing the raw imgaes in obs_decam).

            hchiang2 Hsin-Fang Chiang added a comment - - edited Thank you for your review! Current CameraMapper standardize the input DecoratedImage into Exposure. So I guess when DecoratedImage is actually deprecated ( DM-3190 , DM-3191 ), something will be done with CameraMapper as well. DM-3191 seems to suggest there might be a new type to use. Simply using Exposure in the policy file doesn't work. I will investigate more what else is needed to make it work (probably standardizing the raw imgaes in obs_decam).

            By price,

            DecoratedImage may be deprecated, but it's the only thing we have that fits here. The Exposure constructor will now produce a warning because they are not actually Exposures (e.g., no separate Wcs extension), and plain Image doesn't give us the FITS header.

            I would leave it as DecoratedImage for now.

            If it all looks good, I'll merge it to master.

            hchiang2 Hsin-Fang Chiang added a comment - By price , DecoratedImage may be deprecated, but it's the only thing we have that fits here. The Exposure constructor will now produce a warning because they are not actually Exposures (e.g., no separate Wcs extension), and plain Image doesn't give us the FITS header. I would leave it as DecoratedImage for now. If it all looks good, I'll merge it to master.

            Sounds good. Merge away.

            krughoff Simon Krughoff (Inactive) added a comment - Sounds good. Merge away.

            Merged to master.

             README.md                            |    6 +-
             config/ingest.py                     |    1 +
             decam/camGeom/camera.py              | 2630 +++++++++++++++++-----------------
             policy/DecamMapper.paf               |    6 +-
             python/lsst/obs/decam/decamMapper.py |   16 +-
             python/lsst/obs/decam/ingest.py      |  120 +-
             tests/getRaw.py                      |   73 +-
             7 files changed, 1435 insertions(+), 1417 deletions(-)
            

            hchiang2 Hsin-Fang Chiang added a comment - Merged to master. README.md | 6 +- config/ingest.py | 1 + decam/camGeom/camera.py | 2630 +++++++++++++++++----------------- policy/DecamMapper.paf | 6 +- python/lsst/obs/decam/decamMapper.py | 16 +- python/lsst/obs/decam/ingest.py | 120 +- tests/getRaw.py | 73 +- 7 files changed, 1435 insertions(+), 1417 deletions(-)

            People

              hchiang2 Hsin-Fang Chiang
              hchiang2 Hsin-Fang Chiang
              Simon Krughoff (Inactive)
              David Nidever [X] (Inactive), Hsin-Fang Chiang, Simon Krughoff (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.