Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-17940

Reprocess RC2 with w_2019_10

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Reprocess the HSC RC2 dataset, as defined in DM-11345, using Stack w_2019_10. Steps include makeSkyMap.py, singleFrameDriver.py, mosaic.py, skyCorrection.py, coaddDriver.py, and multiBandDriver.py

        Attachments

          Issue Links

            Activity

            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            Before IHS-1942 and DM-18147 are done, link to the new Goulding S18A BrightObjectMasks  at /scratch/pprice/GouldingMasksS18A.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - Before IHS-1942 and DM-18147 are done, link to the new Goulding S18A BrightObjectMasks  at  /scratch/pprice/GouldingMasksS18A .
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            As IHS-1942 is still pending, the link to BrightObjectMasks is created in the rerun repo: /datasets/hsc/repo/rerun/RC/w_2019_10/DM-17940-sfm/deepCoadd/BrightObjectMasks -> /scratch/pprice/GouldingMasksS18A
             

             

            Show
            hchiang2 Hsin-Fang Chiang added a comment - As IHS-1942 is still pending, the link to BrightObjectMasks is created in the rerun repo: /datasets/hsc/repo/rerun/RC/w_2019_10/ DM-17940 -sfm/deepCoadd/BrightObjectMasks -> /scratch/pprice/GouldingMasksS18A    
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            processCcd/singleFrameDriver for all visits/ccds finished running. 

            (The DM-17843 fix has been merged since the last run.)

            Two "errors" are seen, though processCcd did finish fine in both cases.

            • --id visit=38950 ccd=64; this is DM-15121

              lsst::pex::exceptions::LengthError: 'Box2I(Point2I(470,-1),lsst::geom::Extent2I(21,21)) doesn't fit in image 2048x4176 {0}; Extracting image of PSF candidate {1}'
              

            • --id visit=1872 ccd=11

              processCcd.calibrate.astrometry.matcher INFO: Matched 25 sources
              processCcd.calibrate.astrometry.matcher WARN: Number of matches is smaller than request
              processCcd.calibrate.astrometry.matcher INFO: Purged 661 sources, leaving 113 good sources
              processCcd.calibrate.astrometry.matcher INFO: Matched 1 sources
              processCcd.calibrate.astrometry.matcher WARN: Number of matches is smaller than request
              processCcd.calibrate.astrometry INFO: Fit WCS iter 2 failed; using previous iteration: 
                File "src/sip/CreateWcsWithSip.cc", line 131, in lsst::meas::astrom::sip::CreateWcsWithSip<MatchT>::CreateWcsWithSip(const std::vector<T>&, const lsst::afw::geom::SkyWcs&, int, const lsst::geom::Box2I&, int) [with MatchT = lsst::afw::table::Match<lsst::afw::table::SimpleRecord, lsst::afw::table::SourceRecord>]
                  Number of matches less than requested sip order {0}
              lsst::pex::exceptions::LengthError: 'Number of matches less than requested sip order'
               
              processCcd.calibrate.astrometry INFO: Matched and fit WCS in 1 iterations; found 25 matches with scatter = 0.028 +- 0.019 arcsec
              

             

            Show
            hchiang2 Hsin-Fang Chiang added a comment - processCcd/singleFrameDriver for all visits/ccds finished running.  (The  DM-17843 fix has been merged since the last run.) Two "errors" are seen, though processCcd did finish fine in both cases. --id visit=38950 ccd=64 ; this is DM-15121 lsst::pex::exceptions::LengthError: 'Box2I(Point2I(470,-1),lsst::geom::Extent2I(21,21)) doesn't fit in image 2048x4176 {0}; Extracting image of PSF candidate {1}' --id visit=1872 ccd=11 processCcd.calibrate.astrometry.matcher INFO: Matched 25 sources processCcd.calibrate.astrometry.matcher WARN: Number of matches is smaller than request processCcd.calibrate.astrometry.matcher INFO: Purged 661 sources, leaving 113 good sources processCcd.calibrate.astrometry.matcher INFO: Matched 1 sources processCcd.calibrate.astrometry.matcher WARN: Number of matches is smaller than request processCcd.calibrate.astrometry INFO: Fit WCS iter 2 failed; using previous iteration: File "src/sip/CreateWcsWithSip.cc", line 131, in lsst::meas::astrom::sip::CreateWcsWithSip<MatchT>::CreateWcsWithSip(const std::vector<T>&, const lsst::afw::geom::SkyWcs&, int, const lsst::geom::Box2I&, int) [with MatchT = lsst::afw::table::Match<lsst::afw::table::SimpleRecord, lsst::afw::table::SourceRecord>] Number of matches less than requested sip order {0} lsst::pex::exceptions::LengthError: 'Number of matches less than requested sip order'   processCcd.calibrate.astrometry INFO: Matched and fit WCS in 1 iterations; found 25 matches with scatter = 0.028 +- 0.019 arcsec  
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            The previous case about the matcher is now in DM-18400

            Show
            hchiang2 Hsin-Fang Chiang added a comment - The previous case about the matcher is now in DM-18400
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            The first attempt of running multiband for the COSMOS tract failed with a timeout; in particular, {'tract': 9813, 'filter': 'HSC-Z', 'patch': '6,3'} is the sluggish patch. Ran that patch individually on one node; the memory usage was about 40% at its highest.

            coadd/multiband: the number of data products and empty patches are the same as w_2018_15/DM-14123 (this comment)

            Show
            hchiang2 Hsin-Fang Chiang added a comment - The first attempt of running multiband for the COSMOS tract failed with a timeout; in particular, { 'tract': 9813, 'filter': 'HSC-Z', 'patch': '6,3' } is the sluggish patch. Ran that patch individually on one node; the memory usage was about 40% at its highest. coadd/multiband: the number of data products and empty patches are the same as w_2018_15/ DM-14123 ( this comment )
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            The logs have been copied to /datasets/hsc/repo/rerun/RC/w_2019_10/DM-17940/logs/

            The pipe_analysis and validate_drp runs are at DM-17939 . The jointcal processing is at DM-17938.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - The logs have been copied to /datasets/hsc/repo/rerun/RC/w_2019_10/ DM-17940 /logs/ The pipe_analysis and validate_drp runs are at DM-17939 . The jointcal processing is at DM-17938 .

              People

              • Assignee:
                hchiang2 Hsin-Fang Chiang
                Reporter:
                hchiang2 Hsin-Fang Chiang
                Watchers:
                Hsin-Fang Chiang
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel