Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-8681

Implement RFC-263: Add DECam COSMOS data to /datasets/decam/

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:

      Description

      Transfer Feb2013 COSMOS data to /datasets/decam/ and butlerize it.

      Run some processing to verify the butler repo is useable.

        Attachments

          Issue Links

            Activity

            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            Ingest/butlerize using stack w_2016_51 with a hack in obs_decam ingestCalibs that ensures the "filter" column in the calibRegistry always be "NONE" for all bias masterCal data.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - Ingest/butlerize using stack w_2016_51 with a hack in obs_decam ingestCalibs that ensures the "filter" column in the calibRegistry always be "NONE" for all bias masterCal data.
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            The 291 visits include 39 Y, 34 u, 21 g, 12 r, 87 z, and 98 i -band. In total there are 291*61 = 17751 images.

            I processed these cosmos data using stack w_2016_52 and the u/hfc/DM-8681 branch in obs_decam; ran pipe_drivers singleFrameDriver task with the reference catalog at /datasets/refcats/astrometry_net_data/2mass.20130404 using the slurm cluster. This produced 12050 calexps. The most common failure was "Unable to match sources" in astrometry (4133), but also got other errors such as too few matches, no viable psf candidates, and so on (I'll leave QA to future tickets as QA is not the goal of this ticket)

            Show
            hchiang2 Hsin-Fang Chiang added a comment - The 291 visits include 39 Y, 34 u, 21 g, 12 r, 87 z, and 98 i -band. In total there are 291*61 = 17751 images. I processed these cosmos data using stack w_2016_52 and the u/hfc/ DM-8681 branch in obs_decam; ran pipe_drivers singleFrameDriver task with the reference catalog at /datasets/refcats/astrometry_net_data/2mass.20130404 using the slurm cluster. This produced 12050 calexps. The most common failure was "Unable to match sources" in astrometry (4133), but also got other errors such as too few matches, no viable psf candidates, and so on (I'll leave QA to future tickets as QA is not the goal of this ticket)
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            I'm done; raw, calib, and repo in /datasets/decam can be locked down.

            Please make /datasets/decam/repo/rerun/private sticky.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - I'm done; raw , calib , and repo in /datasets/decam can be locked down. Please make /datasets/decam/repo/rerun/private sticky.
            Hide
            jalt Jason Alt [X] (Inactive) added a comment -

            Locked down and applied sharing permissions

            Show
            jalt Jason Alt [X] (Inactive) added a comment - Locked down and applied sharing permissions

              People

              Assignee:
              hchiang2 Hsin-Fang Chiang
              Reporter:
              hchiang2 Hsin-Fang Chiang
              Reviewers:
              Jason Alt [X] (Inactive)
              Watchers:
              Hsin-Fang Chiang
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.