Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-8701

Copy and ingest HSC Cosmos data

    Details

      Description

      Once RFC-266 is accepted, /gpfs/fs0/scratch/pprice/UH-Cosmos/*.fits needs to be copied into /datasets/hsc/raw/cosmos and ingested into /datasets/hsc/repo.

        Attachments

          Issue Links

            Activity

            price Paul Price created issue -
            price Paul Price made changes -
            Field Original Value New Value
            Link This issue is blocked by RFC-266 [ RFC-266 ]
            hchiang2 Hsin-Fang Chiang made changes -
            Epic Link DM-8333 [ 27862 ]
            hchiang2 Hsin-Fang Chiang made changes -
            Team Process Middleware [ 10206 ]
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            Jason Alt [X], when ready, please give me (instead of Paul) the permission for adding this dataset. I'll take care of it with help from Paul Price.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - Jason Alt [X] , when ready, please give me (instead of Paul) the permission for adding this dataset. I'll take care of it with help from Paul Price .
            Hide
            jalt Jason Alt [X] (Inactive) added a comment -

            Hsin-Fang Chiang I gave you control of hsc/raw/cosmos, hsc/repo, hsc/repo/_mapper, hsc/repo/registry.sqlite3. I could unlock everything but I'm trying to be safe. Please let me know what additional spaces you need control of.

            Show
            jalt Jason Alt [X] (Inactive) added a comment - Hsin-Fang Chiang I gave you control of hsc/raw/cosmos, hsc/repo, hsc/repo/_mapper, hsc/repo/registry.sqlite3. I could unlock everything but I'm trying to be safe. Please let me know what additional spaces you need control of.
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            Start by copying the fits files over

            Show
            hchiang2 Hsin-Fang Chiang added a comment - Start by copying the fits files over
            hchiang2 Hsin-Fang Chiang made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            Jason Alt [X], may you please also give me permission of /datasets/hsc/repo/COSMOS/? I want to create new folders under it and add links.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - Jason Alt [X] , may you please also give me permission of /datasets/hsc/repo/COSMOS/ ? I want to create new folders under it and add links.
            Hide
            jalt Jason Alt [X] (Inactive) added a comment -

            You now own COSMOS.

            Show
            jalt Jason Alt [X] (Inactive) added a comment - You now own COSMOS.
            tjenness Tim Jenness made changes -
            Link This issue is triggered by RFC-266 [ RFC-266 ]
            tjenness Tim Jenness made changes -
            Link This issue is blocked by RFC-266 [ RFC-266 ]
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            Those 18704 raw files have been copied to /datasets/hsc/raw/cosmos/

            With the shared stack w_2017_3, I have ingested them into the butler repo at /datasets/hsc/repo/ (new sym links are made and items are added to the registry)

            I tried processCcd.py and singleFrameDriver.py with a few randomly picked raw data, and they ran okay. I plan to run pipelines with these COSMOS data using the slurm Verification Cluster later this cycle as an exercise.

            Paul Price, would you want to take a look before Jason locks this down?

            Show
            hchiang2 Hsin-Fang Chiang added a comment - Those 18704 raw files have been copied to /datasets/hsc/raw/cosmos/ With the shared stack w_2017_3, I have ingested them into the butler repo at /datasets/hsc/repo/ (new sym links are made and items are added to the registry) I tried processCcd.py and singleFrameDriver.py with a few randomly picked raw data, and they ran okay. I plan to run pipelines with these COSMOS data using the slurm Verification Cluster later this cycle as an exercise. Paul Price , would you want to take a look before Jason locks this down?
            hchiang2 Hsin-Fang Chiang made changes -
            Link This issue blocks DM-9053 [ DM-9053 ]
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            Jason Alt [X], these folders in /datasets/hsc/ can be locked down once Paul Price is happy with them.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - Jason Alt [X] , these folders in /datasets/hsc/ can be locked down once Paul Price is happy with them.
            hchiang2 Hsin-Fang Chiang made changes -
            Reviewers Jason Alt, Paul Price [ jalt, price ]
            Status In Progress [ 3 ] In Review [ 10004 ]
            hchiang2 Hsin-Fang Chiang made changes -
            Story Points 1
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            The ingestImages log is attached, in case it's useful.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - The ingestImages log is attached, in case it's useful.
            hchiang2 Hsin-Fang Chiang made changes -
            Attachment slurm-33688.out.txt [ 29100 ]
            Hide
            price Paul Price added a comment -

            Looks good to me.

            pprice@lsst-dev01:/datasets/hsc/repo $ registryInfo.py /datasets/hsc/repo/ --field COSMOS -s
            filter  field                expTime visit
            HSC-G   COSMOS                7630.0 19180..19220:2
            HSC-I   COSMOS               21683.9 872..910:2^914..946:2^18184..18224:2^908060^908104^908106..908162:2^908166
            HSC-Y   COSMOS               20460.0 628..670:2^674..680:2^18476..18556:2
            HSC-Z   COSMOS                7230.0 18834..18874:2
            NB0921  COSMOS                2220.0 906120^906130^906132..906140:2
            pprice@lsst-dev01:/datasets/hsc/repo $ python -c 'from lsst.daf.persistence import Butler; print Butler("/datasets/hsc/repo").get("raw", visit=18184, ccd=49).getDimensions()'
            CameraMapper INFO: Unable to locate calibRegistry registry in root: /datasets/hsc/repo/CALIB/calibRegistry.sqlite3
            CameraMapper INFO: Unable to locate calibRegistry registry in current dir: ./calibRegistry.sqlite3
            CameraMapper INFO: Loading Posix registry from /datasets/hsc/repo/CALIB
            (2144, 4241)
            

            Show
            price Paul Price added a comment - Looks good to me. pprice@lsst-dev01:/datasets/hsc/repo $ registryInfo.py /datasets/hsc/repo/ --field COSMOS -s filter field expTime visit HSC-G COSMOS 7630.0 19180..19220:2 HSC-I COSMOS 21683.9 872..910:2^914..946:2^18184..18224:2^908060^908104^908106..908162:2^908166 HSC-Y COSMOS 20460.0 628..670:2^674..680:2^18476..18556:2 HSC-Z COSMOS 7230.0 18834..18874:2 NB0921 COSMOS 2220.0 906120^906130^906132..906140:2 pprice@lsst-dev01:/datasets/hsc/repo $ python -c 'from lsst.daf.persistence import Butler; print Butler("/datasets/hsc/repo").get("raw", visit=18184, ccd=49).getDimensions()' CameraMapper INFO: Unable to locate calibRegistry registry in root: /datasets/hsc/repo/CALIB/calibRegistry.sqlite3 CameraMapper INFO: Unable to locate calibRegistry registry in current dir: ./calibRegistry.sqlite3 CameraMapper INFO: Loading Posix registry from /datasets/hsc/repo/CALIB (2144, 4241)
            price Paul Price made changes -
            Status In Review [ 10004 ] Reviewed [ 10101 ]
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            Ha, I didn't realize there is a registryInfo.py in obs_subaru. That looks super useful!

            And searching on Jira led me to DM-4382.

            Thank you Paul.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - Ha, I didn't realize there is a registryInfo.py in obs_subaru . That looks super useful! And searching on Jira led me to DM-4382 . Thank you Paul.
            Hide
            jalt Jason Alt [X] (Inactive) added a comment -

            Locked down raw/[cosmos], repo/[COSMOS|_mapper|registry.sqlite3]

            raw/newhorizons still awaiting organization.

            Show
            jalt Jason Alt [X] (Inactive) added a comment - Locked down raw/ [cosmos] , repo/ [COSMOS|_mapper|registry.sqlite3] raw/newhorizons still awaiting organization.
            jalt Jason Alt [X] (Inactive) made changes -
            Resolution Done [ 10000 ]
            Status Reviewed [ 10101 ] Done [ 10002 ]
            hchiang2 Hsin-Fang Chiang made changes -
            Watchers Hsin-Fang Chiang, Jason Alt [X], Paul Price [ Hsin-Fang Chiang, Jason Alt [X], Paul Price ] Hsin-Fang Chiang, Paul Price [ Hsin-Fang Chiang, Paul Price ]
            Labels datasets
            frossie Frossie Economou made changes -
            Team Process Middleware [ 10206 ] Data Facility [ 12219 ]

              People

              • Assignee:
                hchiang2 Hsin-Fang Chiang
                Reporter:
                price Paul Price
                Reviewers:
                Jason Alt [X] (Inactive), Paul Price
                Watchers:
                Hsin-Fang Chiang, Paul Price
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel