Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-13533

Sky frame calibration data not available for early 2014

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:

      Description

      We have some early-2014 raw HSC data with no corresponding sky frames available in the calib repo /datasets/hsc/calib/20180117/. In particular, sky frames of HSC-R and HSC-Z for as early as dateObs=2014-03-28 are needed by the HSC "RC2" dataset (DM-11345).

      Two example early visits are visit=1202 (HSC-R ) and visit=1166 (HSC-Z), from SSP_UDEEP_COSMOS, and their dateObs is 2014-03-28

      >>> import lsst.daf.persistence
      >>> butler = lsst.daf.persistence.Butler("/datasets/hsc/repo")
      CameraMapper INFO: Loading exposure registry from /datasets/hsc/repo/registry.sqlite3
      CameraMapper INFO: Loading calib registry from /datasets/hsc/repo/CALIB/calibRegistry.sqlite3
      >>> butler.get("raw", visit=1202, ccd=0)
      <lsst.afw.image.exposure.exposure.ExposureU object at 0x7fb5c35d9960>
      >>> butler.get("sky", visit=1202, ccd=0)
      Traceback (most recent call last):
        File "<stdin>", line 1, in <module>
        File "/ssd/lsstsw/stack3_20171021/stack/miniconda3-4.3.21-10a4fa6/Linux64/daf_persistence/14.0-14-g87d16e8+10/python/lsst/daf/persistence/butler.py", line 1410, in get
          raise NoResults("No locations for get:", datasetType, dataId)
      lsst.daf.persistence.butlerExceptions.NoResults: No locations for get: datasetType:sky dataId:DataId(initialdata={'visit': 1202, 'ccd': 0}, tag=set())
      

        Attachments

          Issue Links

            Activity

            Hide
            price Paul Price added a comment - - edited

            Hacked the calibration registry to explicitly include this data. Modified the calib-20180117.txt log file to note the changes:

            UPDATE 2018-02-12: Hsin-Fang Chiang discovered that the validity ranges for
            the sky frames in HSC-R and HSC-Z do not extend sufficiently early to cover
            some Cosmos data we want to process. Modified the validity ranges for those
            explicitly:
             
            UPDATE sky SET validStart = '2014-01-01' WHERE (filter = "HSC-R" AND calibDate \
            = "2014-11-12") OR (filter = "HSC-Z" AND calibDate = "2014-11-02");
            
            

            I've copied the modified calibRegistry.sqlite3 and calib-20180117.txt to the LSST cluster for Hsin-Fang Chiang to update the existing version there.

            Show
            price Paul Price added a comment - - edited Hacked the calibration registry to explicitly include this data. Modified the calib-20180117.txt log file to note the changes: UPDATE 2018-02-12: Hsin-Fang Chiang discovered that the validity ranges for the sky frames in HSC-R and HSC-Z do not extend sufficiently early to cover some Cosmos data we want to process. Modified the validity ranges for those explicitly:   UPDATE sky SET validStart = '2014-01-01' WHERE (filter = "HSC-R" AND calibDate \ = "2014-11-12") OR (filter = "HSC-Z" AND calibDate = "2014-11-02"); I've copied the modified calibRegistry.sqlite3 and calib-20180117.txt to the LSST cluster for Hsin-Fang Chiang to update the existing version there.
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            Paul Price, for processing PDR1 data with sky corr, we would also need to change the HSC-G band too (visit=1886 from 2014-04-03; current earliest validStart 2014-05-08). Would you mind also updating that?

            Other than that, it should be good for the PDR1 data on /datasets/hsc. There are also commissioning data from 2013 but I guess we can turn off the sky correction if we want to process them.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - Paul Price , for processing PDR1 data with sky corr, we would also need to change the HSC-G band too (visit=1886 from 2014-04-03; current earliest validStart 2014-05-08). Would you mind also updating that? Other than that, it should be good for the PDR1 data on /datasets/hsc. There are also commissioning data from 2013 but I guess we can turn off the sky correction if we want to process them.
            Hide
            price Paul Price added a comment -

            UPDATE sky SET validStart = '2014-01-01' WHERE filter = "HSC-G" AND calibDate = "2014-11-04";
            

            The now-twice-modified calibRegistry.sqlite3 and calib-20180117.txt have been copied to the LSST cluster.

            Show
            price Paul Price added a comment - UPDATE sky SET validStart = '2014-01-01' WHERE filter = "HSC-G" AND calibDate = "2014-11-04"; The now-twice-modified calibRegistry.sqlite3 and calib-20180117.txt have been copied to the LSST cluster.
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            The two new files have been copied to /datasets/hsc/calib/20180117/.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - The two new files have been copied to /datasets/hsc/calib/20180117/ .

              People

              Assignee:
              hchiang2 Hsin-Fang Chiang
              Reporter:
              hchiang2 Hsin-Fang Chiang
              Watchers:
              Hsin-Fang Chiang, Paul Price
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  CI Builds

                  No builds found.