Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-10053

/datasets/decam/calib/cp201302/ became obsolete during w_2017_11 and w_2017_13

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:

      Description

      Running this same test command on lsst-dev

      processCcd.py /datasets/decam/repo/ --calib /datasets/decam/calib/cp201302/ --rerun private/hchiang2/w2017_13 --id  visit=177347 ccdnum=3 -C $OBS_DECAM_DIR/config/processCcdCpIsr.py --config  calibrate.doAstrometry=False calibrate.doPhotoCal=False isr.assembleCcd.setGain=False
      

      works with stack w_2017_11 but fails with w_2017_13. First error:

      processCcd FATAL: Failed on dataId={u'date': '2013-02-12', 'visit': 177347, u'hdu': 36, 'ccdnum': 3, u'filter': 'u'}: Unable to retrieve cpBias for {u'date': '2013-02-12', 'visit': 177347, u'hdu': 36, 'ccdnum': 3, u'filter': 'u'}: Mapped path should not be absolute.
      

        Attachments

          Issue Links

            Activity

            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            The error is from obs.base.Mapping.map

            /datasets/decam/calib/cp201302/calibRegistry.sqlite3 has absolute paths in its path column, for calibrations using path in the template. But now only relative path is accepted (DM-8686). I hope rerunning calibration ingestion from the appropriate folder would just solve the problem. Also it's necessary to remake calibRegistry because of DM-5466. Wait for a weekly with DM-5466 before re-ingesting.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - The error is from obs.base.Mapping.map /datasets/decam/calib/cp201302/calibRegistry.sqlite3 has absolute paths in its path column, for calibrations using path in the template. But now only relative path is accepted ( DM-8686 ). I hope rerunning calibration ingestion from the appropriate folder would just solve the problem. Also it's necessary to remake calibRegistry because of DM-5466 . Wait for a weekly with DM-5466 before re-ingesting.
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            Greg Daues may I have the write permission of /datasets/decam/calib/cp201302/ so to update the registry there? I will replace /datasets/decam/calib/cp201302/calibRegistry.sqlite3 with a new one, and also create new symlinks in that folder (new behaviors of the stack for DECam).

            Show
            hchiang2 Hsin-Fang Chiang added a comment - Greg Daues may I have the write permission of /datasets/decam/calib/cp201302/ so to update the registry there? I will replace /datasets/decam/calib/cp201302/calibRegistry.sqlite3 with a new one, and also create new symlinks in that folder (new behaviors of the stack for DECam).
            Hide
            gdaues Greg Daues added a comment -

            The registry

            /datasets/decam/calib/cp201302/calibRegistry.sqlite3

            and the directories in the tree above it have immutable turned off (they can be edited.) If flat/ and bias/ access are also needed let me know about those.

            Show
            gdaues Greg Daues added a comment - The registry /datasets/decam/calib/cp201302/calibRegistry.sqlite3 and the directories in the tree above it have immutable turned off (they can be edited.) If flat/ and bias/ access are also needed let me know about those.
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            Ran ingestCalibs at /datasets/decam/calib/cp201302 with the stack version w_2017_14. A new calibRegistry.sqlite3 was built.

            The equivalent processing works now:

            processCcd.py /datasets/decam/repo/ --calib /datasets/decam/calib/cp201302/ --rerun private/hchiang2/w2017_14 --id  visit=177347 ccdnum=3 -C $OBS_DECAM_DIR/config/processCcdCpIsr.py --config  calibrate.doAstrometry=False calibrate.doPhotoCal=False  
            

            Show
            hchiang2 Hsin-Fang Chiang added a comment - Ran ingestCalibs at /datasets/decam/calib/cp201302 with the stack version w_2017_14. A new calibRegistry.sqlite3 was built. The equivalent processing works now: processCcd.py /datasets/decam/repo/ --calib /datasets/decam/calib/cp201302/ --rerun private /hchiang2/w2017_14 --id visit= 177347 ccdnum= 3 -C $OBS_DECAM_DIR/config/processCcdCpIsr.py --config calibrate.doAstrometry=False calibrate.doPhotoCal=False
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            Greg Daues I'm done with /datasets/decam/calib/cp201302 and please lock it down. Thank you!

            Show
            hchiang2 Hsin-Fang Chiang added a comment - Greg Daues I'm done with /datasets/decam/calib/cp201302 and please lock it down. Thank you!
            Hide
            gdaues Greg Daues added a comment -

            The new cpBIAS cpFLAT directories, the new calibRegistry.sqlite3 , and the directories in the tree are given group lsst_users and are set to immutable. They are ready for general use.

            Show
            gdaues Greg Daues added a comment - The new cpBIAS cpFLAT directories, the new calibRegistry.sqlite3 , and the directories in the tree are given group lsst_users and are set to immutable. They are ready for general use.
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            Thank you Greg Daues!

            Show
            hchiang2 Hsin-Fang Chiang added a comment - Thank you Greg Daues !

              People

              • Assignee:
                hchiang2 Hsin-Fang Chiang
                Reporter:
                hchiang2 Hsin-Fang Chiang
                Reviewers:
                Greg Daues
                Watchers:
                Greg Daues, Hsin-Fang Chiang
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel