Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-13256

Reprocess RC with w_2018_04

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Reprocess the two WIDE tracts of the HSC RC1 dataset, as defined in DMTR-31 Sect. 4.1 , and the three tracts of the HSC RC2 dataset, as defined in DM-11345, using Stack w_2018_04. Steps include makeSkyMap.py, singleFrameDriver.py, mosaic.py, coaddDriver.py, and multiBandDriver.py (-DM-10129/DM-11020-).

        Attachments

          Issue Links

          relates to

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-13436 Run pipe_analysis scripts with the RC2 data of w_2018_04

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-13517 "IndexError: cannot do a non-empty take from an empty axes" in making coadd

          • Done

          Bug - A problem which impairs or prevents the functions of the product. DM-13396 Fix coadd mask propagation

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-11345 Redefine HSC "RC" dataset for bi-weeklies processing

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-13387 Update calibrations in /dataset/hsc/repo for sky subtraction and transmission curves

          • Done

            Activity

            Ascending order - Click to sort in descending order
            4 older comments
            Hide
            Permalink
            hchiang2 Hsin-Fang Chiang added a comment -

            The w_2018_04 version of coaddDriver is broken:

              File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_drivers/6.0b0-hsc-13-g219e95d+3/python/lsst/pipe/drivers/coaddDriver.py", line 315, in coadd
            		 
                detResults = self.detectCoaddSources.runDetection(coadd, idFactory)
            		 
            TypeError: runDetection() missing 1 required positional argument: 'expId'

            This has been fixed in DM-13396 (this commit) that was merged after w_2018_04.

            For coaddDriver.py and beyond, I use a newer version of pipe_drivers 8ba6355.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - The w_2018_04 version of coaddDriver is broken: File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_drivers/6.0b0-hsc-13-g219e95d+3/python/lsst/pipe/drivers/coaddDriver.py", line 315, in coadd detResults = self.detectCoaddSources.runDetection(coadd, idFactory) TypeError: runDetection() missing 1 required positional argument: 'expId' This has been fixed in DM-13396 ( this commit ) that was merged after w_2018_04. For coaddDriver.py and beyond, I use a newer version of pipe_drivers 8ba6355 .
            Hide
            Permalink
            hchiang2 Hsin-Fang Chiang added a comment - - edited

            The RC2 dataset has 432 visits. 6 CCDs failed processCcd/singleFrameDriver, including 1 failing with "Unable to match sources"

            --id visit=34412 ccd=78

            and 5 with "No matches to use for photocal"

            --id visit=35892 ccd=12
            		 
            --id visit=36216 ccd=6
            		 
            --id visit=27068 ccd=96
            		 
            --id visit=17934 ccd=1
            		 
            --id visit=23596 ccd=6

             
            Previously there was a typo in the visit list: we want 36776 rather than 367726. Therefore an separate singleFrameDriver.py was submitted afterwards to process visit= 36776.

            The RC1 WIDE (tract=8766^8767) has 83 visits. 13 CCDs failed processCcd/singleFrameDriver, including:

            • 4 Unable to match sources
            • 1 No matches to use for photocal
            • 4 No objects passed our cuts for consideration as psf stars
            • 4 'PSF star selector found [123] candidates'
              Their data IDs are:

            --id visit=9868 ccd=76
            		 
            --id visit=11582 ccd=76
            		 
            --id visit=7344 ccd=67
            		 
            --id visit=19468 ccd=69
            		 
            --id visit=6478 ccd=99
            		 
            --id visit=6528 ccd=24
            		 
            --id visit=6528 ccd=59
            		 
            --id visit=6528 ccd=67
            		 
            --id visit=6528 ccd=75
            		 
            --id visit=9708 ccd=99
            		 
            --id visit=9736 ccd=67
            		 
            --id visit=17738 ccd=69
            		 
            --id visit=17750 ccd=58

            Show
            hchiang2 Hsin-Fang Chiang added a comment - - edited The RC2 dataset has 432 visits. 6 CCDs failed processCcd/singleFrameDriver, including 1 failing with "Unable to match sources" --id visit=34412 ccd=78 and 5 with "No matches to use for photocal" --id visit=35892 ccd=12 --id visit=36216 ccd=6 --id visit=27068 ccd=96 --id visit=17934 ccd=1 --id visit=23596 ccd=6   Previously there was a typo in the visit list: we want 36776 rather than 367726. Therefore an separate singleFrameDriver.py was submitted afterwards to process visit= 36776. The RC1 WIDE (tract=8766^8767) has 83 visits. 13 CCDs failed processCcd/singleFrameDriver, including: 4 Unable to match sources 1 No matches to use for photocal 4 No objects passed our cuts for consideration as psf stars 4 'PSF star selector found [123] candidates' Their data IDs are: --id visit=9868 ccd=76 --id visit=11582 ccd=76 --id visit=7344 ccd=67 --id visit=19468 ccd=69 --id visit=6478 ccd=99 --id visit=6528 ccd=24 --id visit=6528 ccd=59 --id visit=6528 ccd=67 --id visit=6528 ccd=75 --id visit=9708 ccd=99 --id visit=9736 ccd=67 --id visit=17738 ccd=69 --id visit=17750 ccd=58
            Hide
            Permalink
            hchiang2 Hsin-Fang Chiang added a comment -

            For the COSMOS tract 9813, the coadd products have 76 patches in HSC-G, 78 in HSC-I, 74 in HSC-R, 78 in HSC-Y, 76 in HSC-Z, and 76 in NB0921; in total 458 patches. The empty patches of tract 9813 are:

            • patch 0,0 HSC-G,HSC-I,HSC-R,HSC-Y,HSC-Z,NB0921
            • patch 0,1 HSC-R
            • patch 0,7 HSC-G,HSC-R,HSC-Z,NB0921
            • patch 0,8 HSC-G,HSC-I,HSC-R,HSC-Y,HSC-Z,NB0921
            • patch 1,8 HSC-G,HSC-I,HSC-R,HSC-Y,HSC-Z,NB0921
            • patch 8,0 HSC-G,HSC-R,HSC-Z
            • patch 8,8 HSC-R,NB0921

            In the four WIDE tracts 9697, 9615, 8766 and 8767, each has all 81 patches of coadd products in all 5 bands.

            The MultiBand products have 81 patches in 4 WIDE tracts, 78 patches in COSMOS tract 9813.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - For the COSMOS tract 9813, the coadd products have 76 patches in HSC-G, 78 in HSC-I, 74 in HSC-R, 78 in HSC-Y, 76 in HSC-Z, and 76 in NB0921; in total 458 patches. The empty patches of tract 9813 are: patch 0,0 HSC-G,HSC-I,HSC-R,HSC-Y,HSC-Z,NB0921 patch 0,1 HSC-R patch 0,7 HSC-G,HSC-R,HSC-Z,NB0921 patch 0,8 HSC-G,HSC-I,HSC-R,HSC-Y,HSC-Z,NB0921 patch 1,8 HSC-G,HSC-I,HSC-R,HSC-Y,HSC-Z,NB0921 patch 8,0 HSC-G,HSC-R,HSC-Z patch 8,8 HSC-R,NB0921 In the four WIDE tracts 9697, 9615, 8766 and 8767, each has all 81 patches of coadd products in all 5 bands. The MultiBand products have 81 patches in 4 WIDE tracts, 78 patches in COSMOS tract 9813.
            Hide
            Permalink
            hchiang2 Hsin-Fang Chiang added a comment -

            No fatal errors were seen in coadd and multiband; there were many warnings. For example, contains [12] large footprints that were not deblended in multiband (Same as before).

            The numbers from grepping "Removing visit" in coadd logs are slightly different from the comment in DM-11345: 

            WIDE_VVDS_9697_G:  16
            		 
            WIDE_VVDS_9697_R:  36
            		 
            WIDE_VVDS_9697_I: 589
            		 
            WIDE_VVDS_9697_Z:  44
            		 
            WIDE_VVDS_9697_Y: 128
            		 
             
            WIDE_GAMA15H_9615_G:  14
            		 
            WIDE_GAMA15H_9615_R: 106
            		 
            WIDE_GAMA15H_9615_I:  48
            		 
            WIDE_GAMA15H_9615_Z:   8
            		 
            WIDE_GAMA15H_9615_Y:  14
            		 
             
            UD_COSMOS_9813_G:      53
            		 
            UD_COSMOS_9813_R:     289
            		 
            UD_COSMOS_9813_I:      78
            		 
            UD_COSMOS_9813_Z:      68
            		 
            UD_COSMOS_9813_Y:      51
            		 
            UD_COSMOS_9813_NB0921: 26

            The output repo is available at /datasets/hsc/repo/rerun/RC/w_2018_04/DM-13256/

            Logs have been copied to /datasets/hsc/repo/rerun/RC/w_2018_04/DM-13256//logs/

            Show
            hchiang2 Hsin-Fang Chiang added a comment - No fatal errors were seen in coadd and multiband; there were many warnings. For example,  contains [12] large footprints that were not deblended in multiband (Same as before). The numbers from grepping "Removing visit" in coadd logs are slightly different from the comment in DM-11345 : WIDE_VVDS_9697_G: 16 WIDE_VVDS_9697_R: 36 WIDE_VVDS_9697_I: 589 WIDE_VVDS_9697_Z: 44 WIDE_VVDS_9697_Y: 128 WIDE_GAMA15H_9615_G: 14 WIDE_GAMA15H_9615_R: 106 WIDE_GAMA15H_9615_I: 48 WIDE_GAMA15H_9615_Z: 8 WIDE_GAMA15H_9615_Y: 14 UD_COSMOS_9813_G: 53 UD_COSMOS_9813_R: 289 UD_COSMOS_9813_I: 78 UD_COSMOS_9813_Z: 68 UD_COSMOS_9813_Y: 51 UD_COSMOS_9813_NB0921: 26 The output repo is available at /datasets/hsc/repo/rerun/RC/w_2018_04/ DM-13256 / Logs have been copied to  /datasets/hsc/repo/rerun/RC/w_2018_04/ DM-13256 //logs/
            Hide
            Permalink
            hchiang2 Hsin-Fang Chiang added a comment -

            Some "WARN IndexError: cannot do a non-empty take from an empty axes." are seen in the edge patches of the COSMOS tract: 

            coaddCG.o108352:62229 WARN  2018-02-01T19:13:52.395-0600 coaddDriver: lsst-verify-worker09:62229: Caught IndexError while detection on DataId(initialdata={'tract': 9813, 'filter': 'HSC-G', 'patch': '0,7'}, tag=set()): cannot do a non-empty take from an empty axes.
            		 
            coaddCG.o108352:62233 WARN  2018-02-01T19:16:29.357-0600 coaddDriver: lsst-verify-worker09:62233: Caught IndexError while detection on DataId(initialdata={'tract': 9813, 'filter': 'HSC-G', 'patch': '1,8'}, tag=set()): cannot do a non-empty take from an empty axes.
            		 
            coaddCI.o108354:117797 WARN  2018-02-01T20:59:20.914-0600 coaddDriver: lsst-verify-worker13:117797: Caught IndexError while detection on DataId(initialdata={'tract': 9813, 'filter': 'HSC-I', 'patch': '1,8'}, tag=set()): cannot do a non-empty take from an empty axes.
            		 
            coaddCY.o108357:165550 WARN  2018-02-02T01:05:03.983-0600 coaddDriver: lsst-verify-worker16:165550: Caught IndexError while detection on DataId(initialdata={'tract': 9813, 'filter': 'HSC-Y', 'patch': '1,8'}, tag=set()): cannot do a non-empty take from an empty axes.
            		 
            coaddCZ.o108355:138048 WARN  2018-02-01T20:43:28.665-0600 coaddDriver: lsst-verify-worker14:138048: Caught IndexError while detection on DataId(initialdata={'tract': 9813, 'filter': 'HSC-Z', 'patch': '0,7'}, tag=set()): cannot do a non-empty take from an empty axes.
            		 
            coaddCZ.o108355:138046 WARN  2018-02-01T20:48:39.860-0600 coaddDriver: lsst-verify-worker14:138046: Caught IndexError while detection on DataId(initialdata={'tract': 9813, 'filter': 'HSC-Z', 'patch': '1,8'}, tag=set()): cannot do a non-empty take from an empty axes.

            DM-13517 is filed.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - Some "WARN IndexError: cannot do a non-empty take from an empty axes." are seen in the edge patches of the COSMOS tract: coaddCG.o108352:62229 WARN 2018-02-01T19:13:52.395-0600 coaddDriver: lsst-verify-worker09:62229: Caught IndexError while detection on DataId(initialdata={'tract': 9813, 'filter': 'HSC-G', 'patch': '0,7'}, tag=set()): cannot do a non-empty take from an empty axes. coaddCG.o108352:62233 WARN 2018-02-01T19:16:29.357-0600 coaddDriver: lsst-verify-worker09:62233: Caught IndexError while detection on DataId(initialdata={'tract': 9813, 'filter': 'HSC-G', 'patch': '1,8'}, tag=set()): cannot do a non-empty take from an empty axes. coaddCI.o108354:117797 WARN 2018-02-01T20:59:20.914-0600 coaddDriver: lsst-verify-worker13:117797: Caught IndexError while detection on DataId(initialdata={'tract': 9813, 'filter': 'HSC-I', 'patch': '1,8'}, tag=set()): cannot do a non-empty take from an empty axes. coaddCY.o108357:165550 WARN 2018-02-02T01:05:03.983-0600 coaddDriver: lsst-verify-worker16:165550: Caught IndexError while detection on DataId(initialdata={'tract': 9813, 'filter': 'HSC-Y', 'patch': '1,8'}, tag=set()): cannot do a non-empty take from an empty axes. coaddCZ.o108355:138048 WARN 2018-02-01T20:43:28.665-0600 coaddDriver: lsst-verify-worker14:138048: Caught IndexError while detection on DataId(initialdata={'tract': 9813, 'filter': 'HSC-Z', 'patch': '0,7'}, tag=set()): cannot do a non-empty take from an empty axes. coaddCZ.o108355:138046 WARN 2018-02-01T20:48:39.860-0600 coaddDriver: lsst-verify-worker14:138046: Caught IndexError while detection on DataId(initialdata={'tract': 9813, 'filter': 'HSC-Z', 'patch': '1,8'}, tag=set()): cannot do a non-empty take from an empty axes. DM-13517 is filed.

              People

              Assignee:
              hchiang2 Hsin-Fang Chiang
              Reporter:
              hchiang2 Hsin-Fang Chiang
              Watchers:
              Hsin-Fang Chiang
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.