Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-13532

Reprocess RC with w_2018_08

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Reprocess the two WIDE tracts of the HSC RC1 dataset, as defined in DMTR-31 Sect. 4.1 , and the three tracts of the HSC RC2 dataset, as defined in DM-11345, using Stack w_2018_08. Steps include makeSkyMap.py, singleFrameDriver.py, mosaic.py, skyCorrection.py, coaddDriver.py, and multiBandDriver.py. 

        Attachments

          Issue Links

          relates to

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-13646 Run pipe_analysis scripts with the RC2 data of w_2018_08

          • Done

            Activity

            Ascending order - Click to sort in descending order
            No builds found.
            hchiang2 Hsin-Fang Chiang created issue -
            hchiang2 Hsin-Fang Chiang made changes -
            Field Original Value New Value
            Epic Link DM-12831 [ 36488 ]
            hchiang2 Hsin-Fang Chiang made changes -
            Story Points 4
            hchiang2 Hsin-Fang Chiang made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            Hide
            Permalink
            hchiang2 Hsin-Fang Chiang added a comment -

            For singleFrameDriver, the failed CCDs are the same as in DM-13256 w_2018_04 this comment: 6 in RC2 and 13 in RC1. 

            Many warnings appear in the logs: numpy RuntimeWarnings (DM-13541), "measurePsf.psfDeterminer: NOT scaling kernelSize by stellar quadrupole moment, but using absolute value", "charImage: Source catalog detected and measured with placeholder or default PSF", "Brighter fatter correction did not converge", "photoCal: Source catalog does not have flux uncertainties; using sqrt(flux)", "calibrate.astrometry.matcher: Number of matches is smaller than request", and so on.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - For singleFrameDriver, the failed CCDs are the same as in DM-13256 w_2018_04 this comment : 6 in RC2 and 13 in RC1.  Many warnings appear in the logs: numpy RuntimeWarnings ( DM-13541 ), "measurePsf.psfDeterminer: NOT scaling kernelSize by stellar quadrupole moment, but using absolute value", "charImage: Source catalog detected and measured with placeholder or default PSF", "Brighter fatter correction did not converge", "photoCal: Source catalog does not have flux uncertainties; using sqrt(flux)", "calibrate.astrometry.matcher: Number of matches is smaller than request", and so on.
            Hide
            Permalink
            hchiang2 Hsin-Fang Chiang added a comment -

            For skyCorrection, DM-13548 has been fixed so messages like Unable to fit image for detector "1_47" into image of camera no longer appear. Its products skyCorr for each successful CCDs (in total 53026 = 515*103-19, where the 19 failures are as in the previous comment) and calexp_camera for all 515 visits (432 of RC2 and 83 of RC1) have been generated.

             

             

            Show
            hchiang2 Hsin-Fang Chiang added a comment - For skyCorrection , DM-13548 has been fixed so messages like  Unable to fit image for detector "1_47" into image of camera no longer appear. Its products skyCorr for each successful CCDs (in total 53026 = 515*103-19, where the 19 failures are as in the previous comment) and calexp_camera for all 515 visits (432 of RC2 and 83 of RC1) have been generated.    
            hchiang2 Hsin-Fang Chiang made changes -
            Link This issue relates to DM-13646 [ DM-13646 ]
            Hide
            Permalink
            hchiang2 Hsin-Fang Chiang added a comment - - edited

            As DM-13563/DM-13553 has been fixed, much fewer "IndexError" cannot do a non-empty take from an empty axes are seen now.  15 WARN "Unable to rescale variance of warp (cannot do a non-empty take from an empty axes.); leaving it as-is" and 5 Unable to rescale variance of warp (name 'maskVal' is not defined); leaving it as-is are seen (DM-13553).

            Six "IndexError" are still seen in the edge patches of the COSMOS tract, same as in w_2018_04/DM-13256:

            coaddCG.o110635:60185 WARN  2018-02-25T16:01:07.641-0600 coaddDriver: lsst-verify-worker07:60185: Caught IndexError while detection on DataId(initialdata={'tract': 9813, 'filter': 'HSC-G', 'patch': '0,7'}, tag=set()): cannot do a non-empty take from an empty axes.
            		 
            coaddCG.o110635:60196 WARN  2018-02-25T16:03:28.105-0600 coaddDriver: lsst-verify-worker07:60196: Caught IndexError while detection on DataId(initialdata={'tract': 9813, 'filter': 'HSC-G', 'patch': '1,8'}, tag=set()): cannot do a non-empty take from an empty axes.
            		 
            coaddCI.o110637:47535 WARN  2018-02-25T18:41:00.902-0600 coaddDriver: lsst-verify-worker09:47535: Caught IndexError while detection on DataId(initialdata={'tract': 9813, 'filter': 'HSC-I', 'patch': '1,8'}, tag=set()): cannot do a non-empty take from an empty axes.
            		 
            coaddCY.o110640:58914 WARN  2018-02-26T00:46:07.353-0600 coaddDriver: lsst-verify-worker12:58914: Caught IndexError while detection on DataId(initialdata={'tract': 9813, 'filter': 'HSC-Y', 'patch': '1,8'}, tag=set()): cannot do a non-empty take from an empty axes.
            		 
            coaddCZ.o110638:18303 WARN  2018-02-25T18:17:18.887-0600 coaddDriver: lsst-verify-worker10:18303: Caught IndexError while detection on DataId(initialdata={'tract': 9813, 'filter': 'HSC-Z', 'patch': '0,7'}, tag=set()): cannot do a non-empty take from an empty axes.
            		 
            coaddCZ.o110638:18307 WARN  2018-02-25T18:23:29.164-0600 coaddDriver: lsst-verify-worker10:18307: Caught IndexError while detection on DataId(initialdata={'tract': 9813, 'filter': 'HSC-Z', 'patch': '1,8'}, tag=set()): cannot do a non-empty take from an empty axes.

            This is DM-13517.

            Besides, lots of warnings still present.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - - edited As DM-13563 / DM-13553 has been fixed, much fewer "IndexError" cannot do a non-empty take from an empty axes are seen now.  15 WARN "Unable to rescale variance of warp (cannot do a non-empty take from an empty axes.); leaving it as-is" and 5 Unable to rescale variance of warp (name 'maskVal' is not defined); leaving it as-is are seen ( DM-13553 ). Six "IndexError" are still seen in the edge patches of the COSMOS tract, same as in w_2018_04 / DM-13256 : coaddCG.o110635:60185 WARN 2018-02-25T16:01:07.641-0600 coaddDriver: lsst-verify-worker07:60185: Caught IndexError while detection on DataId(initialdata={'tract': 9813, 'filter': 'HSC-G', 'patch': '0,7'}, tag=set()): cannot do a non-empty take from an empty axes. coaddCG.o110635:60196 WARN 2018-02-25T16:03:28.105-0600 coaddDriver: lsst-verify-worker07:60196: Caught IndexError while detection on DataId(initialdata={'tract': 9813, 'filter': 'HSC-G', 'patch': '1,8'}, tag=set()): cannot do a non-empty take from an empty axes. coaddCI.o110637:47535 WARN 2018-02-25T18:41:00.902-0600 coaddDriver: lsst-verify-worker09:47535: Caught IndexError while detection on DataId(initialdata={'tract': 9813, 'filter': 'HSC-I', 'patch': '1,8'}, tag=set()): cannot do a non-empty take from an empty axes. coaddCY.o110640:58914 WARN 2018-02-26T00:46:07.353-0600 coaddDriver: lsst-verify-worker12:58914: Caught IndexError while detection on DataId(initialdata={'tract': 9813, 'filter': 'HSC-Y', 'patch': '1,8'}, tag=set()): cannot do a non-empty take from an empty axes. coaddCZ.o110638:18303 WARN 2018-02-25T18:17:18.887-0600 coaddDriver: lsst-verify-worker10:18303: Caught IndexError while detection on DataId(initialdata={'tract': 9813, 'filter': 'HSC-Z', 'patch': '0,7'}, tag=set()): cannot do a non-empty take from an empty axes. coaddCZ.o110638:18307 WARN 2018-02-25T18:23:29.164-0600 coaddDriver: lsst-verify-worker10:18307: Caught IndexError while detection on DataId(initialdata={'tract': 9813, 'filter': 'HSC-Z', 'patch': '1,8'}, tag=set()): cannot do a non-empty take from an empty axes. This is DM-13517 . Besides, lots of warnings still present.
            Hide
            Permalink
            hchiang2 Hsin-Fang Chiang added a comment -

            For the COSMOS tract 9813, the coadd products have 76 patches in HSC-G, 78 in HSC-I, 74 in HSC-R, 78 in HSC-Y, 76 in HSC-Z, and 76 in NB0921; in total 458 patches. The empty patches of tract 9813 are:

            • patch 0,0 HSC-G,HSC-I,HSC-R,HSC-Y,HSC-Z,NB0921
            • patch 0,1 HSC-R
            • patch 0,7 HSC-G,HSC-R,HSC-Z,NB0921
            • patch 0,8 HSC-G,HSC-I,HSC-R,HSC-Y,HSC-Z,NB0921
            • patch 1,8 HSC-G,HSC-I,HSC-R,HSC-Y,HSC-Z,NB0921
            • patch 8,0 HSC-G,HSC-R,HSC-Z
            • patch 8,8 HSC-R,NB0921

            In the four WIDE tracts 9697, 9615, 8766 and 8767, each has all 81 patches of coadd products in all 5 bands.

            The MultiBand products have 81 patches in 4 WIDE tracts, 78 patches in COSMOS tract 9813.

            The processing time for MultiBand, especially for the COSMOS tract, is getting noticeably longer than previous runs.

            The output repo is at /datasets/hsc/repo/rerun/RC/w_2018_08/DM-13532.
            The logs have been copied to /datasets/hsc/repo/rerun/RC/w_2018_08/DM-13532/logs.

            pipe_analysis scripts were run for RC2; see DM-13646.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - For the COSMOS tract 9813, the coadd products have 76 patches in HSC-G, 78 in HSC-I, 74 in HSC-R, 78 in HSC-Y, 76 in HSC-Z, and 76 in NB0921; in total 458 patches. The empty patches of tract 9813 are: patch 0,0 HSC-G,HSC-I,HSC-R,HSC-Y,HSC-Z,NB0921 patch 0,1 HSC-R patch 0,7 HSC-G,HSC-R,HSC-Z,NB0921 patch 0,8 HSC-G,HSC-I,HSC-R,HSC-Y,HSC-Z,NB0921 patch 1,8 HSC-G,HSC-I,HSC-R,HSC-Y,HSC-Z,NB0921 patch 8,0 HSC-G,HSC-R,HSC-Z patch 8,8 HSC-R,NB0921 In the four WIDE tracts 9697, 9615, 8766 and 8767, each has all 81 patches of coadd products in all 5 bands. The MultiBand products have 81 patches in 4 WIDE tracts, 78 patches in COSMOS tract 9813. The processing time for MultiBand, especially for the COSMOS tract, is getting noticeably longer than previous runs. The output repo is at /datasets/hsc/repo/rerun/RC/w_2018_08/ DM-13532 . The logs have been copied to /datasets/hsc/repo/rerun/RC/w_2018_08/ DM-13532 /logs . pipe_analysis scripts were run for RC2; see DM-13646 .
            Hide
            Permalink
            hchiang2 Hsin-Fang Chiang added a comment -

            The multiband processing for the COSMOS tract was not finished in one go and --reuse-outputs-from all was used in the second submission; both logs have been included.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - The multiband processing for the COSMOS tract was not finished in one go and --reuse-outputs-from all was used in the second submission; both logs have been included.
            hchiang2 Hsin-Fang Chiang made changes -
            Resolution Done [ 10000 ]
            Status In Progress [ 3 ] Done [ 10002 ]

              People

              Assignee:
              hchiang2 Hsin-Fang Chiang
              Reporter:
              hchiang2 Hsin-Fang Chiang
              Watchers:
              Hsin-Fang Chiang
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.