Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-12929

Reprocess RC with w_2017_50

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Reprocess the HSC RC dataset, as defined in DMTR-31 Sect. 4.1 , using Stack w_2017_50. Steps include makeSkyMap.py, singleFrameDriver.py, mosaic.py, coaddDriver.py, and multiBandDriver.py (DM-10129/DM-11020).

        Attachments

          Issue Links

            Activity

            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            SingleFrameDriver/prcessCcd failed in 11 CCDs at WIDE, and 20 CCDs at COSMOS. Their data IDs are:

            --id visit=278 ccd=95 --id visit=280 ccd=22 --id visit=280 ccd=69 --id visit=280 ccd=103 --id visit=284 ccd=61 --id visit=11698 ccd=68 --id visit=17934 ccd=1 --id visit=23596 ccd=6 --id visit=23600 ccd=1 --id visit=28376 ccd=69 --id visit=28382 ccd=101 --id visit=28392 ccd=101 --id visit=28396 ccd=102 --id visit=28398 ccd=95 --id visit=28398 ccd=101 --id visit=28400 ccd=53 --id visit=28400 ccd=61 --id visit=28400 ccd=95 --id visit=28400 ccd=100 --id visit=28400 ccd=101 --id visit=9868 ccd=76 --id visit=11582 ccd=76 --id visit=7344 ccd=67 --id visit=19468 ccd=69 --id visit=6478 ccd=99 --id visit=6528 ccd=24 --id visit=6528 ccd=67 --id visit=9708 ccd=99 --id visit=9736 ccd=67 --id visit=17738 ccd=69 --id visit=17750 ccd=58
            

            Out of the 31 singleFrameDriver/prcessCcd fatal failures:

            • 5 failed with "Unable to match sources"
            • 4 failed with "No objects passed our cuts for consideration as psf stars"
            • 4 failed with "RuntimeError: No matches to use for photocal"
            • 18 failed with "PSF star selector found [123] candidates"

            The difference from w_2017_48 (DM-12811) is one fewer "RuntimeError: No matches to use for photocal"; --id visit=6528 ccd=59 failed with w_2017_48 but passed with w_2017_50.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - SingleFrameDriver/prcessCcd failed in 11 CCDs at WIDE, and 20 CCDs at COSMOS. Their data IDs are: --id visit= 278 ccd= 95 --id visit= 280 ccd= 22 --id visit= 280 ccd= 69 --id visit= 280 ccd= 103 --id visit= 284 ccd= 61 --id visit= 11698 ccd= 68 --id visit= 17934 ccd= 1 --id visit= 23596 ccd= 6 --id visit= 23600 ccd= 1 --id visit= 28376 ccd= 69 --id visit= 28382 ccd= 101 --id visit= 28392 ccd= 101 --id visit= 28396 ccd= 102 --id visit= 28398 ccd= 95 --id visit= 28398 ccd= 101 --id visit= 28400 ccd= 53 --id visit= 28400 ccd= 61 --id visit= 28400 ccd= 95 --id visit= 28400 ccd= 100 --id visit= 28400 ccd= 101 --id visit= 9868 ccd= 76 --id visit= 11582 ccd= 76 --id visit= 7344 ccd= 67 --id visit= 19468 ccd= 69 --id visit= 6478 ccd= 99 --id visit= 6528 ccd= 24 --id visit= 6528 ccd= 67 --id visit= 9708 ccd= 99 --id visit= 9736 ccd= 67 --id visit= 17738 ccd= 69 --id visit= 17750 ccd= 58 Out of the 31 singleFrameDriver/prcessCcd fatal failures: 5 failed with "Unable to match sources" 4 failed with "No objects passed our cuts for consideration as psf stars" 4 failed with "RuntimeError: No matches to use for photocal" 18 failed with "PSF star selector found [123] candidates" The difference from w_2017_48 ( DM-12811 ) is one fewer "RuntimeError: No matches to use for photocal"; --id visit=6528 ccd=59 failed with w_2017_48 but passed with w_2017_50.
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            For --id visit=6528 ccd=59, one difference between _48 and _50 in the logs is:

            _48:

            processCcd.calibrate.photoCal.match.sourceSelection INFO: Selected 63/705 sources
            processCcd.calibrate.photoCal.match.referenceSelection INFO: Selected 401/5000 references
            processCcd.calibrate.photoCal.match INFO: Matched 0 from 63/705 input and 401/5000 reference sources
            processCcd.calibrate.photoCal.reserve INFO: Reserved 0/0 sources
            processCcd FATAL: Failed on dataId={'visit': 6528, 'ccd': 59, 'field': 'SSP_WIDE', 'dateObs': '2014-09-18', 'pointing': 991, 'filter': 'HSC-Y', 'taiObs': '2014-09-18', 'expTime': 200.0}: RuntimeError: No matches to use for photocal
            

            _50:

            processCcd.calibrate.photoCal.match.sourceSelection INFO: Selected 532/705 sources
            processCcd.calibrate.photoCal.match.referenceSelection INFO: Selected 401/5000 references
            processCcd.calibrate.photoCal.match INFO: Matched 1 from 532/705 input and 401/5000 reference sources
            processCcd.calibrate.photoCal.reserve INFO: Reserved 0/1 sources
            processCcd.calibrate.photoCal INFO: Applying color terms for filterName='y', config.photoCatName=ps1_pv3_3pi_20170110 because config.applyColorTerms is True
            processCcd.calibrate.photoCal WARN: PhotoCal.getZeroPoint: no good stars remain on first iteration; using average of all calibration stars
            processCcd.calibrate.photoCal INFO: Magnitude zero point: 33.446964 +/- 0.000000 from 1 stars
            

            John Swinbank mentioned DM-12798 might sound relevant; Paul Price clarified that new feature was not activated.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - For --id visit=6528 ccd=59 , one difference between _48 and _50 in the logs is: _48: processCcd.calibrate.photoCal.match.sourceSelection INFO: Selected 63 / 705 sources processCcd.calibrate.photoCal.match.referenceSelection INFO: Selected 401 / 5000 references processCcd.calibrate.photoCal.match INFO: Matched 0 from 63 / 705 input and 401 / 5000 reference sources processCcd.calibrate.photoCal.reserve INFO: Reserved 0 / 0 sources processCcd FATAL: Failed on dataId={ 'visit' : 6528 , 'ccd' : 59 , 'field' : 'SSP_WIDE' , 'dateObs' : '2014-09-18' , 'pointing' : 991 , 'filter' : 'HSC-Y' , 'taiObs' : '2014-09-18' , 'expTime' : 200.0 }: RuntimeError: No matches to use for photocal _50: processCcd.calibrate.photoCal.match.sourceSelection INFO: Selected 532 / 705 sources processCcd.calibrate.photoCal.match.referenceSelection INFO: Selected 401 / 5000 references processCcd.calibrate.photoCal.match INFO: Matched 1 from 532 / 705 input and 401 / 5000 reference sources processCcd.calibrate.photoCal.reserve INFO: Reserved 0 / 1 sources processCcd.calibrate.photoCal INFO: Applying color terms for filterName= 'y' , config.photoCatName=ps1_pv3_3pi_20170110 because config.applyColorTerms is True processCcd.calibrate.photoCal WARN: PhotoCal.getZeroPoint: no good stars remain on first iteration; using average of all calibration stars processCcd.calibrate.photoCal INFO: Magnitude zero point: 33.446964 +/- 0.000000 from 1 stars John Swinbank mentioned DM-12798 might sound relevant; Paul Price clarified that new feature was not activated.
            Hide
            price Paul Price added a comment -

            Despite the fact that it got through in the most recent processing run, I think there's still something wrong: it only matched a single source from the reference catalog.

            Show
            price Paul Price added a comment - Despite the fact that it got through in the most recent processing run, I think there's still something wrong: it only matched a single source from the reference catalog.
            Hide
            price Paul Price added a comment -

            After seeing the image (thanks, Hsin-Fang Chiang!), we can understand why it's causing trouble: lots of optical ghosts. It's also in the area where the "Eye of Y" might also have an effect.
            Matching a single object in photoCal is surprising and disappointing, but recent changes (DM-12207) mean that single object should be solid. Maybe the photoCal will produce a slightly errant zeropoint with that single object, but meas_mosaic should be able to clean up. I think this change is something we can be happy about and not worry about.

            Show
            price Paul Price added a comment - After seeing the image (thanks, Hsin-Fang Chiang !), we can understand why it's causing trouble: lots of optical ghosts. It's also in the area where the "Eye of Y" might also have an effect. Matching a single object in photoCal is surprising and disappointing, but recent changes ( DM-12207 ) mean that single object should be solid. Maybe the photoCal will produce a slightly errant zeropoint with that single object, but meas_mosaic should be able to clean up. I think this change is something we can be happy about and not worry about.
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            The meas_mosaic version is with the DM-13062 fix (9d98bb1f) built with w_2017_50.

            In COSMOS tract 9813, the coadd products have 78 patches in HSC-G, 79 in HSC-I, 74 in HSC-R, 79 in HSC-Y, 79 in HSC-Z, and 76 in NB0921; in total 465 patches. The empty patches of tract 9813 are:

            • patch 0,0 HSC-G,HSC-I,HSC-R,HSC-Y,HSC-Z,NB0921
            • patch 0,8 HSC-G,HSC-I,HSC-R,HSC-Y,HSC-Z,NB0921
            • patch 0,1 HSC-R
            • patch 0,7 HSC-R,NB0921
            • patch 1,8 HSC-R,NB0921
            • patch 8,0 HSC-G,HSC-R
            • patch 8,8 HSC-R,NB0921

            In WIDE, both tract 8766 and 8767 have all 81 patches of coadd products in all 5 bands.

            The MultiBand products have 81 patches in WIDE tracts 8766 and 8767, 79 patches in COSMOS tract 9813.

            This is the same as w_2017_48 (DM-12811).

            Show
            hchiang2 Hsin-Fang Chiang added a comment - The meas_mosaic version is with the DM-13062 fix ( 9d98bb1f ) built with w_2017_50 . In COSMOS tract 9813, the coadd products have 78 patches in HSC-G, 79 in HSC-I, 74 in HSC-R, 79 in HSC-Y, 79 in HSC-Z, and 76 in NB0921; in total 465 patches. The empty patches of tract 9813 are: patch 0,0 HSC-G,HSC-I,HSC-R,HSC-Y,HSC-Z,NB0921 patch 0,8 HSC-G,HSC-I,HSC-R,HSC-Y,HSC-Z,NB0921 patch 0,1 HSC-R patch 0,7 HSC-R,NB0921 patch 1,8 HSC-R,NB0921 patch 8,0 HSC-G,HSC-R patch 8,8 HSC-R,NB0921 In WIDE, both tract 8766 and 8767 have all 81 patches of coadd products in all 5 bands. The MultiBand products have 81 patches in WIDE tracts 8766 and 8767, 79 patches in COSMOS tract 9813. This is the same as w_2017_48 ( DM-12811 ).
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            The output repo is available at /datasets/hsc/repo/rerun/RC/w_2017_50/DM-12929/

            Logs have been copied to /datasets/hsc/repo/rerun/RC/w_2017_50/DM-12929/logs

            Show
            hchiang2 Hsin-Fang Chiang added a comment - The output repo is available at /datasets/hsc/repo/rerun/RC/w_2017_50/ DM-12929 / Logs have been copied to /datasets/hsc/repo/rerun/RC/w_2017_50/ DM-12929 /logs
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            Attached is a screen shot of the produced calexp of visit=6528 ccd=59

            Show
            hchiang2 Hsin-Fang Chiang added a comment - Attached is a screen shot of the produced calexp of visit=6528 ccd=59

              People

              • Assignee:
                hchiang2 Hsin-Fang Chiang
                Reporter:
                hchiang2 Hsin-Fang Chiang
                Watchers:
                Hsin-Fang Chiang, Paul Price
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: