Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-9053

Run singleFrameDriver with the HSC COSMOS dataset on Verification Cluster

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      As an exercise, run the single frame processing code (as-in from pipe_drivers) with the HSC COSMOS dataset on the Verification Cluster. If appropriate, develop scripts to facilitate the process.

        Attachments

          Issue Links

            Activity

            hchiang2 Hsin-Fang Chiang created issue -
            hchiang2 Hsin-Fang Chiang made changes -
            Field Original Value New Value
            Epic Link DM-8333 [ 27862 ]
            hchiang2 Hsin-Fang Chiang made changes -
            Link This issue is blocked by DM-8701 [ DM-8701 ]
            hchiang2 Hsin-Fang Chiang made changes -
            Link This issue relates to DM-9235 [ DM-9235 ]
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            Started by using the shared stack w_2017_5 and /datasets/refcats/astrometry_net_data/ps1_pv3_3pi_20170110-and. Also using --clobber-config because of the missing libXext on the compute nodes (ncsa jira LSST-853).

            Show
            hchiang2 Hsin-Fang Chiang added a comment - Started by using the shared stack w_2017_5 and /datasets/refcats/astrometry_net_data/ps1_pv3_3pi_20170110-and . Also using --clobber-config because of the missing libXext on the compute nodes (ncsa jira LSST-853).
            hchiang2 Hsin-Fang Chiang made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            The libXext issue got fixed today, so I should run everything without clobber-config.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - The libXext issue got fixed today, so I should run everything without clobber-config.
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            In the last run, many failures came from the non-science CCDs. But data from those CCDs are not needed.

            Submitted another batch of the 167 visits using the shared stack w_2017_8, /datasets/refcats/astrometry_net_data/ps1_pv3_3pi_20170110-and, and considering only ccd=0..103 to exclude the focus chips.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - In the last run, many failures came from the non-science CCDs. But data from those CCDs are not needed. Submitted another batch of the 167 visits using the shared stack w_2017_8, /datasets/refcats/astrometry_net_data/ps1_pv3_3pi_20170110-and , and considering only ccd=0..103 to exclude the focus chips.
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            Those non-science CCDs will be ignored in the new default config, as changed in DM-9577.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - Those non-science CCDs will be ignored in the new default config, as changed in DM-9577 .
            hchiang2 Hsin-Fang Chiang made changes -
            Story Points 3
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            In total we have 167 visits * 104 CCDs each = 17368 subtasks of singleFrameDriver.
            Out of the 17368, 65 CCDs failed; calexp and src outputs were generated for the other 17303 CCDs. Most of the failures (64 out of 65) were ccd=9

            Show
            hchiang2 Hsin-Fang Chiang added a comment - In total we have 167 visits * 104 CCDs each = 17368 subtasks of singleFrameDriver. Out of the 17368, 65 CCDs failed; calexp and src outputs were generated for the other 17303 CCDs. Most of the failures (64 out of 65) were ccd=9
            hchiang2 Hsin-Fang Chiang made changes -
            Resolution Done [ 10000 ]
            Status In Progress [ 3 ] Done [ 10002 ]
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            Output data are at /datasets/hsc/repo/rerun/private/hchiang2/DM-9053

            Show
            hchiang2 Hsin-Fang Chiang added a comment - Output data are at /datasets/hsc/repo/rerun/private/hchiang2/ DM-9053
            frossie Frossie Economou made changes -
            Team Process Middleware [ 10206 ] Data Facility [ 12219 ]

              People

              • Assignee:
                hchiang2 Hsin-Fang Chiang
                Reporter:
                hchiang2 Hsin-Fang Chiang
                Watchers:
                Hsin-Fang Chiang
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel