Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-13890

Reprocess RC2 with w_2018_14

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Reprocess the HSC RC2 dataset, as defined in DM-11345, using Stack w_2018_14. Steps include makeSkyMap.py, singleFrameDriver.py, mosaic.py, skyCorrection.py, coaddDriver.py, and multiBandDriver.py

        Attachments

          Issue Links

            Activity

            No builds found.
            hchiang2 Hsin-Fang Chiang created issue -
            hchiang2 Hsin-Fang Chiang made changes -
            Field Original Value New Value
            Link This issue relates to IHS-915 [ IHS-915 ]
            hchiang2 Hsin-Fang Chiang made changes -
            Epic Link DM-13926 [ 39699 ]
            hchiang2 Hsin-Fang Chiang made changes -
            Link This issue relates to DM-13665 [ DM-13665 ]
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            Since w_2018_13, meas_mosaic has been included in lsst_distrib and therefore installed on the shared stack on lsst machines. Manual pulling and building for meas_mosaic is no longer needed.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - Since w_2018_13 , meas_mosaic has been included in lsst_distrib and therefore installed on the shared stack on lsst machines. Manual pulling and building for meas_mosaic is no longer needed.
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            For singleFrameDriver, the failed CCDs are the same as w_2018_10 in DM-13647, including 7 CCDs in RC2 and 13 CCDs in RC1.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - For singleFrameDriver , the failed CCDs are the same as w_2018_10 in DM-13647 , including 7 CCDs in RC2 and 13 CCDs in RC1.
            hchiang2 Hsin-Fang Chiang made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            Hide
            hchiang2 Hsin-Fang Chiang added a comment - - edited

            In coadd, same as in w_2018_12/DM-13795, one IndexError: cannot do a non-empty take from an empty axes at 'tract': 9813, 'filter': 'HSC-Z', 'patch': '0,7' was seen:  DM-13517. Many warnings as before. Another kind is the numpy RuntimeWarning in DM-13541.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - - edited In coadd, same as in w_2018_12/ DM-13795 , one IndexError: cannot do a non-empty take from an empty axes at 'tract': 9813, 'filter': 'HSC-Z', 'patch': '0,7' was seen:   DM-13517 . Many warnings as before. Another kind is the numpy RuntimeWarning in DM-13541 .
            hchiang2 Hsin-Fang Chiang made changes -
            Link This issue relates to DM-14048 [ DM-14048 ]
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            For the COSMOS tract 9813, the coadd products have 78 patches in HSC-G, 79 in HSC-I, 74 in HSC-R, 79 in HSC-Y, 77 in HSC-Z, and 76 in NB0921; in total 463 patches. The empty patches of tract 9813 are:

            • patch 0,0 HSC-G,HSC-I,HSC-R,HSC-Y,HSC-Z,NB0921
            • patch 0,1 HSC-R
            • patch 0,7 HSC-R,HSC-Z,NB0921
            • patch 0,8 HSC-G,HSC-I,HSC-R,HSC-Y,HSC-Z,NB0921
            • patch 1,8 HSC-R,NB0921
            • patch 8,0 HSC-G,HSC-R,HSC-Z
            • patch 8,8 HSC-R,NB0921

            In the four WIDE tracts 9697, 9615, 8766 and 8767, each has all 81 patches of coadd products in all 5 bands.

            The MultiBand products have 81 patches in 4 WIDE tracts, 79 patches in COSMOS tract 9813.

            The numbers are the same as the w_2018_12 run (DM-13795)

            Show
            hchiang2 Hsin-Fang Chiang added a comment - For the COSMOS tract 9813, the coadd products have 78 patches in HSC-G, 79 in HSC-I, 74 in HSC-R, 79 in HSC-Y, 77 in HSC-Z, and 76 in NB0921; in total 463 patches. The empty patches of tract 9813 are: patch 0,0 HSC-G,HSC-I,HSC-R,HSC-Y,HSC-Z,NB0921 patch 0,1 HSC-R patch 0,7 HSC-R,HSC-Z,NB0921 patch 0,8 HSC-G,HSC-I,HSC-R,HSC-Y,HSC-Z,NB0921 patch 1,8 HSC-R,NB0921 patch 8,0 HSC-G,HSC-R,HSC-Z patch 8,8 HSC-R,NB0921 In the four WIDE tracts 9697, 9615, 8766 and 8767, each has all 81 patches of coadd products in all 5 bands. The MultiBand products have 81 patches in 4 WIDE tracts, 79 patches in COSMOS tract 9813. The numbers are the same as the w_2018_12 run ( DM-13795 )
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            Similar to the last few RC2 runs again, the multiband processing for the COSMOS tract did not finish in one go and was continued with the --reuse-outputs-from all option.  The first two attempts were killed likely because it went out of memory.  

            Show
            hchiang2 Hsin-Fang Chiang added a comment - Similar to the last few RC2 runs again, the multiband processing for the COSMOS tract did not finish in one go and was continued with the --reuse-outputs-from all option.  The first two attempts were killed likely because it went out of memory.  
            hchiang2 Hsin-Fang Chiang made changes -
            Link This issue relates to DM-14047 [ DM-14047 ]
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            The output repo is at /datasets/hsc/repo/rerun/RC/w_2018_14/DM-13890.
            The logs have been copied to /datasets/hsc/repo/rerun/RC/w_2018_14/DM-13890/logs/.

            The pipe_analysis run is in DM-14047.
            The validate_drp run is in DM-14048.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - The output repo is at /datasets/hsc/repo/rerun/RC/w_2018_14/ DM-13890 . The logs have been copied to /datasets/hsc/repo/rerun/RC/w_2018_14/ DM-13890 /logs/ . The pipe_analysis run is in DM-14047 . The validate_drp run is in DM-14048 .
            hchiang2 Hsin-Fang Chiang made changes -
            Resolution Done [ 10000 ]
            Status In Progress [ 3 ] Done [ 10002 ]
            hchiang2 Hsin-Fang Chiang made changes -
            Story Points 3
            hchiang2 Hsin-Fang Chiang made changes -
            Link This issue relates to DM-13665 [ DM-13665 ]

              People

              Assignee:
              hchiang2 Hsin-Fang Chiang
              Reporter:
              hchiang2 Hsin-Fang Chiang
              Watchers:
              Hsin-Fang Chiang
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  CI Builds

                  No builds found.