Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-13435

Reprocess RC with w_2018_06

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Reprocess the two WIDE tracts of the HSC RC1 dataset, as defined in DMTR-31 Sect. 4.1 , and the three tracts of the HSC RC2 dataset, as defined in DM-11345, using Stack w_2018_06. Steps include makeSkyMap.py, singleFrameDriver.py, mosaic.py, skyCorrection.py, coaddDriver.py, and multiBandDriver.py

        Attachments

          Issue Links

            Activity

            Hide
            tmorton Tim Morton added a comment -

            Do you think it would be possible to start running the pipe_analysis scripts with this next upcoming weekly reprocessing?

            Show
            tmorton Tim Morton added a comment - Do you think it would be possible to start running the pipe_analysis scripts with this next upcoming weekly reprocessing?
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            Yes, I plan to do so. In fact, I've started it with w_2018_04: DM-13436

            Show
            hchiang2 Hsin-Fang Chiang added a comment - Yes, I plan to do so. In fact, I've started it with w_2018_04: DM-13436
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            For singleFrameDriver, the failed CCDs are the same as in DM-13256 w_2018_04 this comment.

            Many RuntimeWarnings are still seen

            RuntimeWarning: invalid value encountered in greater
            

            although DM-11776 has fixed some of them. DM-13541 is filed.

            Sky correction has been made required by default since w_2018_05 (https://community.lsst.org/t/hsc-defaults-now-require-sky-correction/2631 )
            skyCorrection.py is run before coaddDriver. skyCorrection of the RC2 COSMOS data in HSC-R and HSC-Z bands failed, because no sky calibration data were available for early 2014 in those bands; this has been reported and fixed in DM-13533.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - For singleFrameDriver, the failed CCDs are the same as in DM-13256 w_2018_04 this comment . Many RuntimeWarnings are still seen RuntimeWarning: invalid value encountered in greater although DM-11776 has fixed some of them. DM-13541 is filed. Sky correction has been made required by default since w_2018_05 ( https://community.lsst.org/t/hsc-defaults-now-require-sky-correction/2631 ) skyCorrection.py is run before coaddDriver . skyCorrection of the RC2 COSMOS data in HSC-R and HSC-Z bands failed, because no sky calibration data were available for early 2014 in those bands; this has been reported and fixed in DM-13533 .
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            In skyCorrection's logs, there are many Unable to fit image for detector "1_47" into image of camera for the focus CCDs and ccd=9. This led to DM-13548.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - In skyCorrection's logs, there are many Unable to fit image for detector "1_47" into image of camera for the focus CCDs and ccd=9. This led to DM-13548 .
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            The per-filter coadd/multiband products have:

            • RC1 tract=8766 and tract=8767
              80 patches in HSC-I and 81 patches in other 4 filters.
              404 patches in total each.
            • RC2 tract=9697
              81 in HSC-G, 77 in HSC-I, 79 in HSC-R/HSC-Y/HSC-Z.
              395 patches in total.
            • RC2 tract=9615
              80 in HSC-G, 78 in HSC-I, 81 in HSC-R/HSC-Y, 80 in HSC-Z
              400 patches in total.
            • RC2 tract=9813
              75 patches in HSC-G, 79 in HSC-I, 74 in HSC-R, 76 in HSC-Y, 73 in HSC-Z, 74 in NB0921.
              Except only 78 patches in HSC-I for deepCoadd_measMatch and deepCoadd_measMatch; they are missing at patch=1,8 additionally.
              451 patches in total, 450 patches for deepCoadd_measMatch and deepCoadd_measMatch.

             The empty patches are:

            • RC1 tract=8766
              • 2,3: HSC-I
            • RC1 tract=8767
              • 3,3: HSC-I
            • RC2 tract=9697
              • 0,0: HSC-I
              • 1,4: HSC-R
              • 2,6: HSC-Y
              • 5,0: HSC-I,HSC-Z
              • 6,0: HSC-I
              • 6,8: HSC-Y
              • 7,0: HSC-I,HSC-Z
              • 8,4: HSC-R
            • RC2 tract=9615
              • 0,8: HSC-I
              • 1,7: HSC-I
              • 5,3: HSC-I
              • 6,4: HSC-G
              • 7,5: HSC-Z
            • RC2 tract=9813
              • 0,0: HSC-G,HSC-I,HSC-R,HSC-Y,HSC-Z,NB0921
              • 0,1: HSC-R
              • 0,6: NB0921
              • 0,7: HSC-G,HSC-R,HSC-Z,NB0921
              • 0,8: HSC-G,HSC-I,HSC-R,HSC-Y,HSC-Z,NB0921
              • 1,8: HSC-G,(HSC-I),HSC-R,HSC-Y,HSC-Z,NB0921
              • 2,0: HSC-G
              • 3,8: HSC-Z
              • 6,8: HSC-Y,HSC-Z,NB0921
              • 7,8: HSC-Z
              • 8,0: HSC-G,HSC-R,HSC-Z
              • 8,1: HSC-Y
              • 8,8: HSC-R,NB0921
                 

            The multiband deepCoadd_mergeDet and deepCoadd_ref products have 81 patches in the 4 WIDE tracts, 79 patches in COSMOS tract=9813.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - The per-filter coadd/multiband products have: RC1 tract=8766 and tract=8767 80 patches in HSC-I and 81 patches in other 4 filters. 404 patches in total each. RC2 tract=9697 81 in HSC-G, 77 in HSC-I, 79 in HSC-R/HSC-Y/HSC-Z. 395 patches in total. RC2 tract=9615 80 in HSC-G, 78 in HSC-I, 81 in HSC-R/HSC-Y, 80 in HSC-Z 400 patches in total. RC2 tract=9813 75 patches in HSC-G, 79 in HSC-I, 74 in HSC-R, 76 in HSC-Y, 73 in HSC-Z, 74 in NB0921. Except only 78 patches in HSC-I for deepCoadd_measMatch and deepCoadd_measMatch ; they are missing at patch=1,8 additionally. 451 patches in total, 450 patches for deepCoadd_measMatch and deepCoadd_measMatch .  The empty patches are: RC1 tract=8766 2,3: HSC-I RC1 tract=8767 3,3: HSC-I RC2 tract=9697 0,0: HSC-I 1,4: HSC-R 2,6: HSC-Y 5,0: HSC-I,HSC-Z 6,0: HSC-I 6,8: HSC-Y 7,0: HSC-I,HSC-Z 8,4: HSC-R RC2 tract=9615 0,8: HSC-I 1,7: HSC-I 5,3: HSC-I 6,4: HSC-G 7,5: HSC-Z RC2 tract=9813 0,0: HSC-G,HSC-I,HSC-R,HSC-Y,HSC-Z,NB0921 0,1: HSC-R 0,6: NB0921 0,7: HSC-G,HSC-R,HSC-Z,NB0921 0,8: HSC-G,HSC-I,HSC-R,HSC-Y,HSC-Z,NB0921 1,8: HSC-G,(HSC-I),HSC-R,HSC-Y,HSC-Z,NB0921 2,0: HSC-G 3,8: HSC-Z 6,8: HSC-Y,HSC-Z,NB0921 7,8: HSC-Z 8,0: HSC-G,HSC-R,HSC-Z 8,1: HSC-Y 8,8: HSC-R,NB0921   The multiband deepCoadd_mergeDet and deepCoadd_ref products have 81 patches in the 4 WIDE tracts, 79 patches in COSMOS tract=9813.
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            Many "IndexError" are seen in coaddDriver logs; DM-13563 accounts for most of the additional empty patches compared to w_2018_04 (1 of tract=8766, 1 of tract=8767, 10 of tract=9697, 5 of tract=9615, 8 of tract=9813). In COSMOS, 5 empty patches close to the edge are not made with the error messages as in DM-13517.   COSMOS deepCoadd_calexp at filter=HSC-I, patch=1,8 was made with w_2018_06, although it failed with w_2018_04 (DM-13256, DM-13517).

            The output repo is available at /datasets/hsc/repo/rerun/RC/w_2018_06/DM-13435/

            Logs have been copied to /datasets/hsc/repo/rerun/RC/w_2018_06/DM-13435/logs/

            pipe_analysis scripts were run as well; see DM-13523.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - Many "IndexError" are seen in coaddDriver logs; DM-13563 accounts for most of the additional empty patches compared to w_2018_04 (1 of tract=8766, 1 of tract=8767, 10 of tract=9697, 5 of tract=9615, 8 of tract=9813). In COSMOS, 5 empty patches close to the edge are not made with the error messages as in DM-13517 .   COSMOS deepCoadd_calexp at filter=HSC-I, patch=1,8 was made with w_2018_06, although it failed with w_2018_04 ( DM-13256 , DM-13517 ). The output repo is available at /datasets/hsc/repo/rerun/RC/w_2018_06/ DM-13435 / Logs have been copied to  /datasets/hsc/repo/rerun/RC/w_2018_06/ DM-13435 /logs/ pipe_analysis scripts were run as well; see DM-13523 .

              People

              • Assignee:
                hchiang2 Hsin-Fang Chiang
                Reporter:
                hchiang2 Hsin-Fang Chiang
                Watchers:
                Hsin-Fang Chiang, Tim Morton
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel