Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-11381

Remove the trailing repo from butler root /datasets/decam/repo

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:

      Description

      As per RFC-308, we will remove the trailing repo from butler root /datasets/decam/repo

        Attachments

          Issue Links

          is blocked by

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-14098 Update /datasets documentation to reflect RFC-308

          • Won't Fix
          is triggered by

          RFC - An issue whose primary purpose is to advertise an intent and allow stakeholders to comment. RFC-308 Remove /repo from paths to datasets

          • Retired

            Activity

            Ascending order - Click to sort in descending order
            Hide
            Permalink
            hchiang2 Hsin-Fang Chiang added a comment -

            doing the change together with the clean-up in DM-8124

            Show
            hchiang2 Hsin-Fang Chiang added a comment - doing the change together with the clean-up in DM-8124
            Hide
            Permalink
            hchiang2 Hsin-Fang Chiang added a comment -

            I'm moving everything not managed by Butler directly into /datasets/decam/_internal/.

            Need to re-ingest master calibs `cp201302` too, because the links need to be updated accordingly.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - I'm moving everything not managed by Butler directly into /datasets/decam/_internal/ . Need to re-ingest master calibs `cp201302` too, because the links need to be updated accordingly.
            Hide
            Permalink
            hchiang2 Hsin-Fang Chiang added a comment -

            Attempting to use w_2017_33 to ingest defects files failed with this command: 

            $ ingestCalibs.py /datasets/decam/ --calib . --validity 0 --calibType defect ../bpmDes/2012-09-01/D_n20130115t0130_c33_r1975p01_bpm.fits
            		 
            root INFO: Loading config overrride file u'/ssd/lsstsw/stack_20170409/Linux64/obs_decam/13.0-19-ga4adf72/config/ingestCalibs.py'
            		 
            CameraMapper INFO: Loading Posix calib registry from /datasets/decam/_internal/calib/cp201302
            		 
            ingestCalibs.parse WARN: Error reading ../bpmDes/2012-09-01/D_n20130115t0130_c33_r1975p01_bpm.fits extensions set(['S9', 'S8', 'S3', 'S2', 'S1', 'S7', 'S6', 'S5', 'S4', 'S30', 'N2', 'S19', 'S18', 'S13', 'S12', 'S11', 'S10', 'S17', 'S16', 'S15', 'S14', 'N3', 'N28', 'N19', 'N23', 'N22', 'N21', 'N20', 'N27', 'N26', 'N25', 'N24', 'N29', 'N13', 'S22', 'S23', 'S20', 'S21', 'S26', 'S27', 'S24', 'S25', 'N1', 'S28', 'S29', 'N4', 'N5', 'N6', 'N7', 'N12', 'N8', 'N10', 'N11', 'N16', 'N17', 'N14', 'N9', 'N18', 'N15', 'N31', 'S31'])
            		 
            Traceback (most recent call last):
            		 
              File "/ssd/lsstsw/stack_20170409/Linux64/pipe_tasks/13.0-44-g89df7c8/bin/ingestCalibs.py", line 3, in <module>
            		 
                IngestCalibsTask.parseAndRun()
            		 
              File "/ssd/lsstsw/stack_20170409/Linux64/pipe_tasks/13.0-44-g89df7c8/python/lsst/pipe/tasks/ingest.py", line 399, in parseAndRun
            		 
                task.run(args)
            		 
              File "/ssd/lsstsw/stack_20170409/Linux64/pipe_tasks/13.0-44-g89df7c8/python/lsst/pipe/tasks/ingestCalibs.py", line 231, in run
            		 
                outfile = self.parse.getDestination(args.butler, fileInfo, infile)
            		 
              File "/ssd/lsstsw/stack_20170409/Linux64/obs_decam/13.0-19-ga4adf72/python/lsst/obs/decam/ingestCalibs.py", line 118, in getDestination
            		 
                calibType = self.getCalibType(filename)
            		 
              File "/ssd/lsstsw/stack_20170409/Linux64/pipe_tasks/13.0-44-g89df7c8/python/lsst/pipe/tasks/ingestCalibs.py", line 32, in getCalibType
            		 
                (filename, self.config.hdu))
            		 
            RuntimeError: Unable to find the required header keyword OBSTYPE in ../bpmDes/2012-09-01/D_n20130115t0130_c33_r1975p01_bpm.fits, hdu 0

            The same command worked with w_2017_27 but failed with w_2017_28. It seemed this was caused by DM-11136 which merged between w_2017_27 and w_2017_28. In particular, this commit changed the default mode of ingestCalibs from "skip" to "move".

            So now we need to specify --mode skip for ingesting defects. Adding it to the command line, ingestCalibs works with the new stack.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - Attempting to use w_2017_33 to ingest defects files failed with this command: $ ingestCalibs.py /datasets/decam/ --calib . --validity 0 --calibType defect ../bpmDes/ 2012 - 09 - 01 /D_n20130115t0130_c33_r1975p01_bpm.fits root INFO: Loading config overrride file u '/ssd/lsstsw/stack_20170409/Linux64/obs_decam/13.0-19-ga4adf72/config/ingestCalibs.py' CameraMapper INFO: Loading Posix calib registry from /datasets/decam/_internal/calib/cp201302 ingestCalibs.parse WARN: Error reading ../bpmDes/ 2012 - 09 - 01 /D_n20130115t0130_c33_r1975p01_bpm.fits extensions set([ 'S9' , 'S8' , 'S3' , 'S2' , 'S1' , 'S7' , 'S6' , 'S5' , 'S4' , 'S30' , 'N2' , 'S19' , 'S18' , 'S13' , 'S12' , 'S11' , 'S10' , 'S17' , 'S16' , 'S15' , 'S14' , 'N3' , 'N28' , 'N19' , 'N23' , 'N22' , 'N21' , 'N20' , 'N27' , 'N26' , 'N25' , 'N24' , 'N29' , 'N13' , 'S22' , 'S23' , 'S20' , 'S21' , 'S26' , 'S27' , 'S24' , 'S25' , 'N1' , 'S28' , 'S29' , 'N4' , 'N5' , 'N6' , 'N7' , 'N12' , 'N8' , 'N10' , 'N11' , 'N16' , 'N17' , 'N14' , 'N9' , 'N18' , 'N15' , 'N31' , 'S31' ]) Traceback (most recent call last): File "/ssd/lsstsw/stack_20170409/Linux64/pipe_tasks/13.0-44-g89df7c8/bin/ingestCalibs.py" , line 3 , in <module> IngestCalibsTask.parseAndRun() File "/ssd/lsstsw/stack_20170409/Linux64/pipe_tasks/13.0-44-g89df7c8/python/lsst/pipe/tasks/ingest.py" , line 399 , in parseAndRun task.run(args) File "/ssd/lsstsw/stack_20170409/Linux64/pipe_tasks/13.0-44-g89df7c8/python/lsst/pipe/tasks/ingestCalibs.py" , line 231 , in run outfile = self.parse.getDestination(args.butler, fileInfo, infile) File "/ssd/lsstsw/stack_20170409/Linux64/obs_decam/13.0-19-ga4adf72/python/lsst/obs/decam/ingestCalibs.py" , line 118 , in getDestination calibType = self.getCalibType(filename) File "/ssd/lsstsw/stack_20170409/Linux64/pipe_tasks/13.0-44-g89df7c8/python/lsst/pipe/tasks/ingestCalibs.py" , line 32 , in getCalibType (filename, self.config.hdu)) RuntimeError: Unable to find the required header keyword OBSTYPE in ../bpmDes/ 2012 - 09 - 01 /D_n20130115t0130_c33_r1975p01_bpm.fits, hdu 0 The same command worked with w_2017_27 but failed with w_2017_28 . It seemed this was caused by DM-11136 which merged between w_2017_27 and w_2017_28 . In particular, this commit changed the default mode of ingestCalibs from "skip" to "move". So now we need to specify --mode skip for ingesting defects. Adding it to the command line, ingestCalibs works with the new stack.
            Hide
            Permalink
            hchiang2 Hsin-Fang Chiang added a comment -

            Re-ingested /datasets/decam/_internal/calib/cp201302/ using w_2017_33 successfully.

            cd /datasets/decam/_internal/calib/cp201302/
            		 
            ingestCalibs.py /datasets/decam/ --calib . --validity 0 --calibType defect --mode skip  ../bpmDes/20*/*fits
            		 
            ingestCalibs.py /datasets/decam/ --calib . --validity 90 --mode=link bias/*fits.fz
            		 
            ingestCalibs.py /datasets/decam/ --calib . --validity 90 --mode=link flat/*fits.fz

            Show
            hchiang2 Hsin-Fang Chiang added a comment - Re-ingested /datasets/decam/_internal/calib/cp201302/ using w_2017_33 successfully. cd /datasets/decam/_internal/calib/cp201302/ ingestCalibs.py /datasets/decam/ --calib . --validity 0 --calibType defect --mode skip ../bpmDes/ 20 */*fits ingestCalibs.py /datasets/decam/ --calib . --validity 90 --mode=link bias/*fits.fz ingestCalibs.py /datasets/decam/ --calib . --validity 90 --mode=link flat/*fits.fz
            Hide
            Permalink
            hchiang2 Hsin-Fang Chiang added a comment -

            Done, together with DM-11317 and DM-8124. Folders/files are locked down.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - Done, together with DM-11317 and DM-8124 . Folders/files are locked down.

              People

              Assignee:
              hchiang2 Hsin-Fang Chiang
              Reporter:
              hchiang2 Hsin-Fang Chiang
              Watchers:
              Hsin-Fang Chiang
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.