Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-8681

Implement RFC-263: Add DECam COSMOS data to /datasets/decam/

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:

      Description

      Transfer Feb2013 COSMOS data to /datasets/decam/ and butlerize it.

      Run some processing to verify the butler repo is useable.

        Attachments

          Issue Links

          is triggered by

          RFC - An issue whose primary purpose is to advertise an intent and allow stakeholders to comment. RFC-263 Add DECam COSMOS data to /datasets/decam/

          • Implemented
          relates to

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-8258 Curate and process some public DECam COSMOS data on Verification Cluster

          • Done

            Activity

            Ascending order - Click to sort in descending order
            No builds found.
            hchiang2 Hsin-Fang Chiang created issue -
            hchiang2 Hsin-Fang Chiang made changes -
            Field Original Value New Value
            Epic Link DM-8333 [ 27862 ]
            hchiang2 Hsin-Fang Chiang made changes -
            Link This issue is triggered by RFC-263 [ RFC-263 ]
            hchiang2 Hsin-Fang Chiang made changes -
            Link This issue relates to DM-8258 [ DM-8258 ]
            hchiang2 Hsin-Fang Chiang made changes -
            Summary Implement RFC-263 Implement RFC-263: Add DECam COSMOS data to /datasets/decam/
            Hide
            Permalink
            hchiang2 Hsin-Fang Chiang added a comment -

            Ingest/butlerize using stack w_2016_51 with a hack in obs_decam ingestCalibs that ensures the "filter" column in the calibRegistry always be "NONE" for all bias masterCal data.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - Ingest/butlerize using stack w_2016_51 with a hack in obs_decam ingestCalibs that ensures the "filter" column in the calibRegistry always be "NONE" for all bias masterCal data.
            hchiang2 Hsin-Fang Chiang made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            Hide
            Permalink
            hchiang2 Hsin-Fang Chiang added a comment -

            The 291 visits include 39 Y, 34 u, 21 g, 12 r, 87 z, and 98 i -band. In total there are 291*61 = 17751 images.

            I processed these cosmos data using stack w_2016_52 and the u/hfc/DM-8681 branch in obs_decam; ran pipe_drivers singleFrameDriver task with the reference catalog at /datasets/refcats/astrometry_net_data/2mass.20130404 using the slurm cluster. This produced 12050 calexps. The most common failure was "Unable to match sources" in astrometry (4133), but also got other errors such as too few matches, no viable psf candidates, and so on (I'll leave QA to future tickets as QA is not the goal of this ticket)

            Show
            hchiang2 Hsin-Fang Chiang added a comment - The 291 visits include 39 Y, 34 u, 21 g, 12 r, 87 z, and 98 i -band. In total there are 291*61 = 17751 images. I processed these cosmos data using stack w_2016_52 and the u/hfc/ DM-8681 branch in obs_decam; ran pipe_drivers singleFrameDriver task with the reference catalog at /datasets/refcats/astrometry_net_data/2mass.20130404 using the slurm cluster. This produced 12050 calexps. The most common failure was "Unable to match sources" in astrometry (4133), but also got other errors such as too few matches, no viable psf candidates, and so on (I'll leave QA to future tickets as QA is not the goal of this ticket)
            hchiang2 Hsin-Fang Chiang made changes -
            Story Points 3
            Hide
            Permalink
            hchiang2 Hsin-Fang Chiang added a comment -

            I'm done; raw, calib, and repo in /datasets/decam can be locked down.

            Please make /datasets/decam/repo/rerun/private sticky.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - I'm done; raw , calib , and repo in /datasets/decam can be locked down. Please make /datasets/decam/repo/rerun/private sticky.
            hchiang2 Hsin-Fang Chiang made changes -
            Reviewers Jason Alt [ jalt ]
            Status In Progress [ 3 ] In Review [ 10004 ]
            Hide
            Permalink
            jalt Jason Alt [X] (Inactive) added a comment -

            Locked down and applied sharing permissions

            Show
            jalt Jason Alt [X] (Inactive) added a comment - Locked down and applied sharing permissions
            jalt Jason Alt [X] (Inactive) made changes -
            Resolution Done [ 10000 ]
            Status In Review [ 10004 ] Done [ 10002 ]
            hchiang2 Hsin-Fang Chiang made changes -
            Watchers Hsin-Fang Chiang, Jason Alt [X] [ Hsin-Fang Chiang, Jason Alt [X] ] Hsin-Fang Chiang [ Hsin-Fang Chiang ]
            Labels datasets
            frossie Frossie Economou made changes -
            Team Process Middleware [ 10206 ] Data Facility [ 12219 ]

              People

              Assignee:
              hchiang2 Hsin-Fang Chiang
              Reporter:
              hchiang2 Hsin-Fang Chiang
              Reviewers:
              Jason Alt [X] (Inactive)
              Watchers:
              Hsin-Fang Chiang
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.