Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-11184

Reprocess RC with w_2017_28

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Reprocess the HSC RC dataset using w_2017_28 following the same steps as in DM-10129/DM-11020

        Attachments

          Issue Links

            Activity

            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            w_2017_28 was not released last weekend so it's not yet available in the shared stack on lsst-dev so far. Therefore I cannot start this run yet. As there is a regularly scheduled maintenance this Thursday this ticket may be further delayed.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - w_2017_28 was not released last weekend so it's not yet available in the shared stack on lsst-dev so far. Therefore I cannot start this run yet. As there is a regularly scheduled maintenance this Thursday this ticket may be further delayed.
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            w_2017_28 just became available on the shared stack. With it I have scons-ed the current mater of meas_mosaic (f4b19d1) and submitted the first batch of jobs. The slurm queue has been occupied so it may be a while before these jobs run.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - w_2017_28 just became available on the shared stack. With it I have scons-ed the current mater of meas_mosaic (f4b19d1) and submitted the first batch of jobs. The slurm queue has been occupied so it may be a while before these jobs run.
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            Hitting IHS-341 again with No space left on device at /project

            Show
            hchiang2 Hsin-Fang Chiang added a comment - Hitting IHS-341 again with No space left on device at /project
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -
            Single Frame Processing

            81 CCDs failed, 62 from COSMOS and 19 from WIDE. The failed data IDs are the same as in w_2017_25 (DM-11020) and w_2017_26 (DM-11165).

            Mosaic

            Completed.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - Single Frame Processing 81 CCDs failed, 62 from COSMOS and 19 from WIDE. The failed data IDs are the same as in w_2017_25 ( DM-11020 ) and w_2017_26 ( DM-11165 ). Mosaic Completed.
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -
            coadd

            In COSMOS tract 9813, the coadd products have 78 patches in HSC-G, 79 in HSC-I, 74 in HSC-R, 79 in HSC-Y, 79 in HSC-Z, and 76 in NB0921; in total 465 patches. The empty patches of tract 9813 are:

            • patch 0,0 HSC-G,HSC-I,HSC-R,HSC-Y,HSC-Z,NB0921
            • patch 0,8 HSC-G,HSC-I,HSC-R,HSC-Y,HSC-Z,NB0921
            • patch 0,1 HSC-R
            • patch 0,7 HSC-R,NB0921
            • patch 1,8 HSC-R,NB0921
            • patch 8,0 HSC-G,HSC-R
            • patch 8,8 HSC-R,NB0921

            In WIDE, both tract 8766 and 8767 have all 81 patches of coadd products in all 5 bands.

            Compared with the coadd by w_2017_26 (DM-11165), some empty patches are now filled. Much more calexps are getting selected for making coadds; in other words, fewer calexps are being removed by PsfWcsSelectImagesTask. This is expected by the merge of DM-11243.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - coadd In COSMOS tract 9813, the coadd products have 78 patches in HSC-G, 79 in HSC-I, 74 in HSC-R, 79 in HSC-Y, 79 in HSC-Z, and 76 in NB0921; in total 465 patches. The empty patches of tract 9813 are: patch 0,0 HSC-G,HSC-I,HSC-R,HSC-Y,HSC-Z,NB0921 patch 0,8 HSC-G,HSC-I,HSC-R,HSC-Y,HSC-Z,NB0921 patch 0,1 HSC-R patch 0,7 HSC-R,NB0921 patch 1,8 HSC-R,NB0921 patch 8,0 HSC-G,HSC-R patch 8,8 HSC-R,NB0921 In WIDE, both tract 8766 and 8767 have all 81 patches of coadd products in all 5 bands. Compared with the coadd by w_2017_26 ( DM-11165 ), some empty patches are now filled. Much more calexps are getting selected for making coadds; in other words, fewer calexps are being removed by PsfWcsSelectImagesTask . This is expected by the merge of DM-11243 .
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -
            MULTIBAND

            Completed: 81 patches in WIDE tracts 8766 and 8767, 79 patches in COSMOS tract 9813.

            Because the inode shortage at /project earlier this week, jobs starting with mosaic were run and outputted to /scratch space. A job to sync up the repos is pending in the lsstvc slurm queue.

            Logs have been copied to /project/hsc_rc/w_2017_28/DM-11184/logs/

            Show
            hchiang2 Hsin-Fang Chiang added a comment - MULTIBAND Completed: 81 patches in WIDE tracts 8766 and 8767, 79 patches in COSMOS tract 9813. Because the inode shortage at /project earlier this week, jobs starting with mosaic were run and outputted to /scratch space. A job to sync up the repos is pending in the lsstvc slurm queue. Logs have been copied to /project/hsc_rc/w_2017_28/ DM-11184 /logs/
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            Syncing has finished.

            The nominal repo path is /datasets/hsc/repo/rerun/RC/w_2017_28/DM-11184/
            (realpath: /project/hsc_rc/w_2017_28/DM-11184)

            Show
            hchiang2 Hsin-Fang Chiang added a comment - Syncing has finished. The nominal repo path is /datasets/hsc/repo/rerun/RC/w_2017_28/ DM-11184 / (realpath: /project/hsc_rc/w_2017_28/ DM-11184 )
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            With w_2017_28 the mosaic jobs had many warnings about the missed matches, e.g.

            Persisted match record with ID 111331498155159102 not found in catalog 1
            

            This has been reported in DM-11356.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - With w_2017_28 the mosaic jobs had many warnings about the missed matches, e.g. Persisted match record with ID 111331498155159102 not found in catalog 1 This has been reported in DM-11356 .

              People

              • Assignee:
                hchiang2 Hsin-Fang Chiang
                Reporter:
                hchiang2 Hsin-Fang Chiang
                Watchers:
                Hsin-Fang Chiang
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: