Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-8681

Implement RFC-263: Add DECam COSMOS data to /datasets/decam/

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:

      Description

      Transfer Feb2013 COSMOS data to /datasets/decam/ and butlerize it.

      Run some processing to verify the butler repo is useable.

        Attachments

          Issue Links

            Activity

            hchiang2 Hsin-Fang Chiang created issue -
            hchiang2 Hsin-Fang Chiang made changes -
            Field Original Value New Value
            Epic Link DM-8333 [ 27862 ]
            hchiang2 Hsin-Fang Chiang made changes -
            Link This issue is triggered by RFC-263 [ RFC-263 ]
            hchiang2 Hsin-Fang Chiang made changes -
            Link This issue relates to DM-8258 [ DM-8258 ]
            hchiang2 Hsin-Fang Chiang made changes -
            Summary Implement RFC-263 Implement RFC-263: Add DECam COSMOS data to /datasets/decam/
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            Ingest/butlerize using stack w_2016_51 with a hack in obs_decam ingestCalibs that ensures the "filter" column in the calibRegistry always be "NONE" for all bias masterCal data.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - Ingest/butlerize using stack w_2016_51 with a hack in obs_decam ingestCalibs that ensures the "filter" column in the calibRegistry always be "NONE" for all bias masterCal data.
            hchiang2 Hsin-Fang Chiang made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            The 291 visits include 39 Y, 34 u, 21 g, 12 r, 87 z, and 98 i -band. In total there are 291*61 = 17751 images.

            I processed these cosmos data using stack w_2016_52 and the u/hfc/DM-8681 branch in obs_decam; ran pipe_drivers singleFrameDriver task with the reference catalog at /datasets/refcats/astrometry_net_data/2mass.20130404 using the slurm cluster. This produced 12050 calexps. The most common failure was "Unable to match sources" in astrometry (4133), but also got other errors such as too few matches, no viable psf candidates, and so on (I'll leave QA to future tickets as QA is not the goal of this ticket)

            Show
            hchiang2 Hsin-Fang Chiang added a comment - The 291 visits include 39 Y, 34 u, 21 g, 12 r, 87 z, and 98 i -band. In total there are 291*61 = 17751 images. I processed these cosmos data using stack w_2016_52 and the u/hfc/ DM-8681 branch in obs_decam; ran pipe_drivers singleFrameDriver task with the reference catalog at /datasets/refcats/astrometry_net_data/2mass.20130404 using the slurm cluster. This produced 12050 calexps. The most common failure was "Unable to match sources" in astrometry (4133), but also got other errors such as too few matches, no viable psf candidates, and so on (I'll leave QA to future tickets as QA is not the goal of this ticket)
            hchiang2 Hsin-Fang Chiang made changes -
            Story Points 3
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            I'm done; raw, calib, and repo in /datasets/decam can be locked down.

            Please make /datasets/decam/repo/rerun/private sticky.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - I'm done; raw , calib , and repo in /datasets/decam can be locked down. Please make /datasets/decam/repo/rerun/private sticky.
            hchiang2 Hsin-Fang Chiang made changes -
            Reviewers Jason Alt [ jalt ]
            Status In Progress [ 3 ] In Review [ 10004 ]
            Hide
            jalt Jason Alt [X] (Inactive) added a comment -

            Locked down and applied sharing permissions

            Show
            jalt Jason Alt [X] (Inactive) added a comment - Locked down and applied sharing permissions
            jalt Jason Alt [X] (Inactive) made changes -
            Resolution Done [ 10000 ]
            Status In Review [ 10004 ] Done [ 10002 ]
            hchiang2 Hsin-Fang Chiang made changes -
            Watchers Hsin-Fang Chiang, Jason Alt [X] [ Hsin-Fang Chiang, Jason Alt [X] ] Hsin-Fang Chiang [ Hsin-Fang Chiang ]
            Labels datasets
            frossie Frossie Economou made changes -
            Team Process Middleware [ 10206 ] Data Facility [ 12219 ]

              People

              • Assignee:
                hchiang2 Hsin-Fang Chiang
                Reporter:
                hchiang2 Hsin-Fang Chiang
                Reviewers:
                Jason Alt [X] (Inactive)
                Watchers:
                Hsin-Fang Chiang
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel