Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-13647

Reprocess RC2 with w_2018_10

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Reprocess the HSC RC2 dataset, as defined in DM-11345, using Stack w_2018_10. Steps include makeSkyMap.py, singleFrameDriver.py, mosaic.py, skyCorrection.py, coaddDriver.py, and multiBandDriver.py. 

        Attachments

          Issue Links

          relates to

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-13761 Run pipe_analysis scripts with the RC2 data of w_2018_10

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-13804 Run validate_drp with w_2018_10 HSC RC2 outputs

          • Done

            Activity

            Ascending order - Click to sort in descending order
            Hide
            Permalink
            hchiang2 Hsin-Fang Chiang added a comment - - edited

            For the 432 visits of the RC2 dataset, 7 CCDs failed processCcd/singleFrameDriver, including 1 failing with "Unable to match sources"

            --id visit=34412 ccd=78

            and 6 with "No matches to use for photocal"

            --id visit=35892 ccd=12
            		 
            --id visit=36216 ccd=6
            		 
            --id visit=27068 ccd=96
            		 
            --id visit=19712 ccd=33
            		 
            --id visit=17934 ccd=1
            		 
            --id visit=23596 ccd=6

            Compared with w_2018_08/DM-13532, visit=19712 ccd=33 is new; it passed with w_2018_09 but failed with w_2018_10.
             

            For the 83 visits of RC1 WIDE (tract=8766^8767), 13 CCDs failed processCcd/singleFrameDriver, same as in w_2018_04 this comment

            --id visit=9868 ccd=76
            		 
            --id visit=11582 ccd=76
            		 
            --id visit=7344 ccd=67
            		 
            --id visit=19468 ccd=69
            		 
            --id visit=6478 ccd=99
            		 
            --id visit=6528 ccd=24
            		 
            --id visit=6528 ccd=59
            		 
            --id visit=6528 ccd=67
            		 
            --id visit=6528 ccd=75
            		 
            --id visit=9708 ccd=99
            		 
            --id visit=9736 ccd=67
            		 
            --id visit=17738 ccd=69
            		 
            --id visit=17750 ccd=58

            Many numpy RuntimeWarnings are seen in the logs: DM-13541.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - - edited For the 432 visits of the RC2 dataset, 7 CCDs failed processCcd/singleFrameDriver, including 1 failing with "Unable to match sources" --id visit=34412 ccd=78 and 6 with "No matches to use for photocal" --id visit=35892 ccd=12 --id visit=36216 ccd=6 --id visit=27068 ccd=96 --id visit=19712 ccd=33 --id visit=17934 ccd=1 --id visit=23596 ccd=6 Compared with w_2018_08 / DM-13532 , visit=19712 ccd=33 is new; it passed with w_2018_09 but failed with w_2018_10.   For the 83 visits of RC1 WIDE (tract=8766^8767), 13 CCDs failed processCcd/singleFrameDriver, same as in w_2018_04 this comment --id visit=9868 ccd=76 --id visit=11582 ccd=76 --id visit=7344 ccd=67 --id visit=19468 ccd=69 --id visit=6478 ccd=99 --id visit=6528 ccd=24 --id visit=6528 ccd=59 --id visit=6528 ccd=67 --id visit=6528 ccd=75 --id visit=9708 ccd=99 --id visit=9736 ccd=67 --id visit=17738 ccd=69 --id visit=17750 ccd=58 Many numpy RuntimeWarnings are seen in the logs: DM-13541 .
            Hide
            Permalink
            hchiang2 Hsin-Fang Chiang added a comment -

            visit=19712 ccd=33 processCcd changed between w_2018_09 and w_2018_10

            Last lines of logs in w_2018_10:

            processCcd.calibrate.astrometry INFO: Matched and fit WCS in 3 iterations; found 41 matches with scatter = 1.084 +- 0.465 arcsec
            		 
            processCcd.calibrate.photoCal.match.sourceSelection INFO: Selected 2279/2560 sources
            		 
            processCcd.calibrate.photoCal.match.referenceSelection INFO: Selected 307/2533 references
            		 
            processCcd.calibrate.photoCal.match INFO: Matched 0 from 2279/2560 input and 307/2533 reference sources
            		 
            processCcd.calibrate.photoCal.reserve INFO: Reserved 0/0 sources
            		 
            processCcd FATAL: Failed on dataId={'visit': 19712, 'ccd': 33, 'field': 'SSP_UDEEP_COSMOS', 'dateObs': '2015-01-21', 'pointing': 1116, 'filter': 'HSC-I', 'taiObs': '2015-01-21', 'expTime': 300.0}: RuntimeError: No matches to use for photocal

            In w_2018_09:

            processCcd.calibrate.astrometry INFO: Matched and fit WCS in 3 iterations; found 338 matches with scatter = 0.080 +- 0.061 arcsec
            		 
            processCcd.calibrate.photoCal.match.sourceSelection INFO: Selected 2389/2679 sources
            		 
            processCcd.calibrate.photoCal.match.referenceSelection INFO: Selected 300/2509 references
            		 
            processCcd.calibrate.photoCal.match INFO: Matched 133 from 2389/2679 input and 300/2509 reference sources

            Defects of ccd=33 changed in DM-13731 and might be relevant .

            Show
            hchiang2 Hsin-Fang Chiang added a comment - visit=19712 ccd=33  processCcd changed between w_2018_09 and w_2018_10 Last lines of logs in w_2018_10: processCcd.calibrate.astrometry INFO: Matched and fit WCS in 3 iterations; found 41 matches with scatter = 1.084 +- 0.465 arcsec processCcd.calibrate.photoCal.match.sourceSelection INFO: Selected 2279/2560 sources processCcd.calibrate.photoCal.match.referenceSelection INFO: Selected 307/2533 references processCcd.calibrate.photoCal.match INFO: Matched 0 from 2279/2560 input and 307/2533 reference sources processCcd.calibrate.photoCal.reserve INFO: Reserved 0/0 sources processCcd FATAL: Failed on dataId={'visit': 19712, 'ccd': 33, 'field': 'SSP_UDEEP_COSMOS', 'dateObs': '2015-01-21', 'pointing': 1116, 'filter': 'HSC-I', 'taiObs': '2015-01-21', 'expTime': 300.0}: RuntimeError: No matches to use for photocal In w_2018_09: processCcd.calibrate.astrometry INFO: Matched and fit WCS in 3 iterations; found 338 matches with scatter = 0.080 +- 0.061 arcsec processCcd.calibrate.photoCal.match.sourceSelection INFO: Selected 2389/2679 sources processCcd.calibrate.photoCal.match.referenceSelection INFO: Selected 300/2509 references processCcd.calibrate.photoCal.match INFO: Matched 133 from 2389/2679 input and 300/2509 reference sources Defects of ccd=33 changed in DM-13731 and might be relevant .
            Hide
            Permalink
            hchiang2 Hsin-Fang Chiang added a comment -

            For the COSMOS tract 9813, the coadd products have 76 patches in HSC-G, 78 in HSC-I, 74 in HSC-R, 78 in HSC-Y, 76 in HSC-Z, and 76 in NB0921; in total 458 patches. The empty patches of tract 9813 are:

            • patch 0,0 HSC-G,HSC-I,HSC-R,HSC-Y,HSC-Z,NB0921
            • patch 0,1 HSC-R
            • patch 0,7 HSC-G,HSC-R,HSC-Z,NB0921
            • patch 0,8 HSC-G,HSC-I,HSC-R,HSC-Y,HSC-Z,NB0921
            • patch 1,8 HSC-G,HSC-I,HSC-R,HSC-Y,HSC-Z,NB0921
            • patch 8,0 HSC-G,HSC-R,HSC-Z
            • patch 8,8 HSC-R,NB0921

            In the four WIDE tracts 9697, 9615, 8766 and 8767, each has all 81 patches of coadd products in all 5 bands.

            The MultiBand products have 81 patches in 4 WIDE tracts, 78 patches in COSMOS tract 9813.

            This is the same as w_2018_08 DM-13532 (this comment)

            Show
            hchiang2 Hsin-Fang Chiang added a comment - For the COSMOS tract 9813, the coadd products have 76 patches in HSC-G, 78 in HSC-I, 74 in HSC-R, 78 in HSC-Y, 76 in HSC-Z, and 76 in NB0921; in total 458 patches. The empty patches of tract 9813 are: patch 0,0 HSC-G,HSC-I,HSC-R,HSC-Y,HSC-Z,NB0921 patch 0,1 HSC-R patch 0,7 HSC-G,HSC-R,HSC-Z,NB0921 patch 0,8 HSC-G,HSC-I,HSC-R,HSC-Y,HSC-Z,NB0921 patch 1,8 HSC-G,HSC-I,HSC-R,HSC-Y,HSC-Z,NB0921 patch 8,0 HSC-G,HSC-R,HSC-Z patch 8,8 HSC-R,NB0921 In the four WIDE tracts 9697, 9615, 8766 and 8767, each has all 81 patches of coadd products in all 5 bands. The MultiBand products have 81 patches in 4 WIDE tracts, 78 patches in COSMOS tract 9813. This is the same as w_2018_08 DM-13532 ( this comment )
            Hide
            Permalink
            hchiang2 Hsin-Fang Chiang added a comment -

            Six "IndexError" (DM-13517) are still seen in the edge patches of the COSMOS tract, as in w_2018_08/DM-13532

            coaddCG.o118272:23899 WARN  2018-03-11T04:57:43.384-0500 coaddDriver: lsst-verify-worker27:23899: Caught IndexError while detection on DataId(initialdata={'tract': 9813, 'filter': 'HSC-G', 'patch': '0,7'}, tag=set()): cannot do a non-empty take from an empty axes.
            		 
            coaddCG.o118272:23908 WARN  2018-03-11T05:00:11.662-0500 coaddDriver: lsst-verify-worker27:23908: Caught IndexError while detection on DataId(initialdata={'tract': 9813, 'filter': 'HSC-G', 'patch': '1,8'}, tag=set()): cannot do a non-empty take from an empty axes.
            		 
            coaddCI.o118274:159018 WARN  2018-03-11T06:52:13.584-0500 coaddDriver: lsst-verify-worker42:159018: Caught IndexError while detection on DataId(initialdata={'tract': 9813, 'filter': 'HSC-I', 'patch': '1,8'}, tag=set()): cannot do a non-empty take from an empty axes.
            		 
            coaddCY.o118277:36746 WARN  2018-03-11T12:04:21.467-0500 coaddDriver: lsst-verify-worker44:36746: Caught IndexError while detection on DataId(initialdata={'tract': 9813, 'filter': 'HSC-Y', 'patch': '1,8'}, tag=set()): cannot do a non-empty take from an empty axes.
            		 
            coaddCZ.o118275:59915 WARN  2018-03-11T06:36:49.739-0500 coaddDriver: lsst-verify-worker14:59915: Caught IndexError while detection on DataId(initialdata={'tract': 9813, 'filter': 'HSC-Z', 'patch': '0,7'}, tag=set()): cannot do a non-empty take from an empty axes.
            		 
            coaddCZ.o118275:59914 WARN  2018-03-11T06:40:21.330-0500 coaddDriver: lsst-verify-worker14:59914: Caught IndexError while detection on DataId(initialdata={'tract': 9813, 'filter': 'HSC-Z', 'patch': '1,8'}, tag=set()): cannot do a non-empty take from an empty axes.

             

             

            Show
            hchiang2 Hsin-Fang Chiang added a comment - Six "IndexError" ( DM-13517 ) are still seen in the edge patches of the COSMOS tract, as in w_2018_08/ DM-13532 coaddCG.o118272:23899 WARN 2018-03-11T04:57:43.384-0500 coaddDriver: lsst-verify-worker27:23899: Caught IndexError while detection on DataId(initialdata={'tract': 9813, 'filter': 'HSC-G', 'patch': '0,7'}, tag=set()): cannot do a non-empty take from an empty axes. coaddCG.o118272:23908 WARN 2018-03-11T05:00:11.662-0500 coaddDriver: lsst-verify-worker27:23908: Caught IndexError while detection on DataId(initialdata={'tract': 9813, 'filter': 'HSC-G', 'patch': '1,8'}, tag=set()): cannot do a non-empty take from an empty axes. coaddCI.o118274:159018 WARN 2018-03-11T06:52:13.584-0500 coaddDriver: lsst-verify-worker42:159018: Caught IndexError while detection on DataId(initialdata={'tract': 9813, 'filter': 'HSC-I', 'patch': '1,8'}, tag=set()): cannot do a non-empty take from an empty axes. coaddCY.o118277:36746 WARN 2018-03-11T12:04:21.467-0500 coaddDriver: lsst-verify-worker44:36746: Caught IndexError while detection on DataId(initialdata={'tract': 9813, 'filter': 'HSC-Y', 'patch': '1,8'}, tag=set()): cannot do a non-empty take from an empty axes. coaddCZ.o118275:59915 WARN 2018-03-11T06:36:49.739-0500 coaddDriver: lsst-verify-worker14:59915: Caught IndexError while detection on DataId(initialdata={'tract': 9813, 'filter': 'HSC-Z', 'patch': '0,7'}, tag=set()): cannot do a non-empty take from an empty axes. coaddCZ.o118275:59914 WARN 2018-03-11T06:40:21.330-0500 coaddDriver: lsst-verify-worker14:59914: Caught IndexError while detection on DataId(initialdata={'tract': 9813, 'filter': 'HSC-Z', 'patch': '1,8'}, tag=set()): cannot do a non-empty take from an empty axes.    
            Hide
            Permalink
            hchiang2 Hsin-Fang Chiang added a comment -

            The output repo is at /datasets/hsc/repo/rerun/RC/w_2018_10/DM-13647/.
            The logs have been copied to /datasets/hsc/repo/rerun/RC/w_2018_10/DM-13647/logs/.

            The pipe_analysis run is in DM-13761.
            The validate_drp run is in DM-13804.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - The output repo is at /datasets/hsc/repo/rerun/RC/w_2018_10/ DM-13647 / . The logs have been copied to /datasets/hsc/repo/rerun/RC/w_2018_10/ DM-13647 /logs/ . The pipe_analysis run is in DM-13761 . The validate_drp run is in DM-13804 .

              People

              Assignee:
              hchiang2 Hsin-Fang Chiang
              Reporter:
              hchiang2 Hsin-Fang Chiang
              Watchers:
              Hsin-Fang Chiang
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.