Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-8701

Copy and ingest HSC Cosmos data

    XMLWordPrintable

    Details

      Description

      Once RFC-266 is accepted, /gpfs/fs0/scratch/pprice/UH-Cosmos/*.fits needs to be copied into /datasets/hsc/raw/cosmos and ingested into /datasets/hsc/repo.

        Attachments

          Issue Links

          blocks

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-9053 Run singleFrameDriver with the HSC COSMOS dataset on Verification Cluster

          • Done
          is triggered by

          RFC - An issue whose primary purpose is to advertise an intent and allow stakeholders to comment. RFC-266 Incorporate HSC Cosmos data into /datasets

          • Implemented

            Activity

            Ascending order - Click to sort in descending order
            6 older comments
            Hide
            Permalink
            hchiang2 Hsin-Fang Chiang added a comment -

            Jason Alt [X], these folders in /datasets/hsc/ can be locked down once Paul Price is happy with them.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - Jason Alt [X] , these folders in /datasets/hsc/ can be locked down once Paul Price is happy with them.
            Hide
            Permalink
            hchiang2 Hsin-Fang Chiang added a comment -

            The ingestImages log is attached, in case it's useful.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - The ingestImages log is attached, in case it's useful.
            Hide
            Permalink
            price Paul Price added a comment -

            Looks good to me.

            pprice@lsst-dev01:/datasets/hsc/repo $ registryInfo.py /datasets/hsc/repo/ --field COSMOS -s
            		 
            filter  field                expTime visit
            		 
            HSC-G   COSMOS                7630.0 19180..19220:2
            		 
            HSC-I   COSMOS               21683.9 872..910:2^914..946:2^18184..18224:2^908060^908104^908106..908162:2^908166
            		 
            HSC-Y   COSMOS               20460.0 628..670:2^674..680:2^18476..18556:2
            		 
            HSC-Z   COSMOS                7230.0 18834..18874:2
            		 
            NB0921  COSMOS                2220.0 906120^906130^906132..906140:2
            		 
            pprice@lsst-dev01:/datasets/hsc/repo $ python -c 'from lsst.daf.persistence import Butler; print Butler("/datasets/hsc/repo").get("raw", visit=18184, ccd=49).getDimensions()'
            		 
            CameraMapper INFO: Unable to locate calibRegistry registry in root: /datasets/hsc/repo/CALIB/calibRegistry.sqlite3
            		 
            CameraMapper INFO: Unable to locate calibRegistry registry in current dir: ./calibRegistry.sqlite3
            		 
            CameraMapper INFO: Loading Posix registry from /datasets/hsc/repo/CALIB
            		 
            (2144, 4241)

            Show
            price Paul Price added a comment - Looks good to me. pprice@lsst-dev01:/datasets/hsc/repo $ registryInfo.py /datasets/hsc/repo/ --field COSMOS -s filter field expTime visit HSC-G COSMOS 7630.0 19180..19220:2 HSC-I COSMOS 21683.9 872..910:2^914..946:2^18184..18224:2^908060^908104^908106..908162:2^908166 HSC-Y COSMOS 20460.0 628..670:2^674..680:2^18476..18556:2 HSC-Z COSMOS 7230.0 18834..18874:2 NB0921 COSMOS 2220.0 906120^906130^906132..906140:2 pprice@lsst-dev01:/datasets/hsc/repo $ python -c 'from lsst.daf.persistence import Butler; print Butler("/datasets/hsc/repo").get("raw", visit=18184, ccd=49).getDimensions()' CameraMapper INFO: Unable to locate calibRegistry registry in root: /datasets/hsc/repo/CALIB/calibRegistry.sqlite3 CameraMapper INFO: Unable to locate calibRegistry registry in current dir: ./calibRegistry.sqlite3 CameraMapper INFO: Loading Posix registry from /datasets/hsc/repo/CALIB (2144, 4241)
            Hide
            Permalink
            hchiang2 Hsin-Fang Chiang added a comment -

            Ha, I didn't realize there is a registryInfo.py in obs_subaru. That looks super useful!

            And searching on Jira led me to DM-4382.

            Thank you Paul.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - Ha, I didn't realize there is a registryInfo.py in obs_subaru . That looks super useful! And searching on Jira led me to DM-4382 . Thank you Paul.
            Hide
            Permalink
            jalt Jason Alt [X] (Inactive) added a comment -

            Locked down raw/[cosmos], repo/[COSMOS|_mapper|registry.sqlite3]

            raw/newhorizons still awaiting organization.

            Show
            jalt Jason Alt [X] (Inactive) added a comment - Locked down raw/ [cosmos] , repo/ [COSMOS|_mapper|registry.sqlite3] raw/newhorizons still awaiting organization.

              People

              Assignee:
              hchiang2 Hsin-Fang Chiang
              Reporter:
              price Paul Price
              Reviewers:
              Jason Alt [X] (Inactive), Paul Price
              Watchers:
              Hsin-Fang Chiang, Paul Price
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.