Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-17400

Reprocess RC2 with w_2019_06

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Reprocess the HSC RC2 dataset, as defined in DM-11345, using Stack w_2019_06. Steps include makeSkyMap.py, singleFrameDriver.py, mosaic.py, skyCorrection.py, coaddDriver.py, and multiBandDriver.py. 

        Attachments

          Issue Links

          relates to

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-17401 Run pipe_analysis & validate_drp with w_2019_06 HSC RC2

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-17402 Run jointcal with HSC-RC2 and w_2019_06

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-17843 A new set of processCcd failure in HSC-RC2 reprocessing

          • Done

            Activity

            Ascending order - Click to sort in descending order
            Hide
            Permalink
            hchiang2 Hsin-Fang Chiang added a comment -

            For the 432 visits of the RC2 dataset, 60 CCDs failed processCcd/singleFrameDriver with "Unable to match sources". Their data IDs are:

            --id visit=34338 ccd=75
            		 
            --id visit=34362 ccd=38
            		 
            --id visit=34382 ccd=43
            		 
            --id visit=34384 ccd=88
            		 
            --id visit=34402 ccd=61
            		 
            --id visit=34422 ccd=42
            		 
            --id visit=34424 ccd=2
            		 
            --id visit=34448 ccd=79
            		 
            --id visit=34450 ccd=25
            		 
            --id visit=34478 ccd=21
            		 
            --id visit=34482 ccd=50
            		 
            --id visit=34644 ccd=72
            		 
            --id visit=34690 ccd=39
            		 
            --id visit=36140 ccd=30
            		 
            --id visit=36170 ccd=18
            		 
            --id visit=36182 ccd=80
            		 
            --id visit=36212 ccd=34
            		 
            --id visit=36234 ccd=82
            		 
            --id visit=36240 ccd=0
            		 
            --id visit=36258 ccd=73
            		 
            --id visit=36428 ccd=61
            		 
            --id visit=36432 ccd=23
            		 
            --id visit=36492 ccd=50
            		 
            --id visit=34942 ccd=44
            		 
            --id visit=36762 ccd=18
            		 
            --id visit=36774 ccd=24
            		 
            --id visit=36792 ccd=34
            		 
            --id visit=36808 ccd=26
            		 
            --id visit=36828 ccd=16
            		 
            --id visit=36828 ccd=73
            		 
            --id visit=26046 ccd=32
            		 
            --id visit=26050 ccd=74
            		 
            --id visit=26058 ccd=34
            		 
            --id visit=26060 ccd=70
            		 
            -id visit=26072 ccd=12
            		 
            --id visit=26080 ccd=66
            		 
            --id visit=26084 ccd=64
            		 
            --id visit=23864 ccd=12
            		 
            --id visit=1308 ccd=40
            		 
            --id visit=23224 ccd=59
            		 
            --id visit=23232 ccd=32
            		 
            --id visit=27106 ccd=33
            		 
            --id visit=27106 ccd=41
            		 
            --id visit=27128 ccd=26
            		 
            --id visit=27134 ccd=28
            		 
            --id visit=440 ccd=92
            		 
            --id visit=452 ccd=35
            		 
            --id visit=452 ccd=42
            		 
            --id visit=472 ccd=30
            		 
            --id visit=1246 ccd=19
            		 
            --id visit=19696 ccd=20
            		 
            --id visit=30500 ccd=20
            		 
            --id visit=1180 ccd=89
            		 
            --id visit=1184 ccd=26
            		 
            --id visit=1194 ccd=26
            		 
            --id visit=360 ccd=26
            		 
            --id visit=22628 ccd=18
            		 
            --id visit=22664 ccd=26
            		 
            --id visit=23046 ccd=26
            		 
            --id visit=23050 ccd=84

            This looks related to the astrometry matcher default change to PessimisticB by DM-14857, which got merged before w_2019_04. DM-17843 has been filed in case further investigation is needed. There is a jump in many metrics shown in squash.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - For the 432 visits of the RC2 dataset, 60 CCDs failed processCcd/singleFrameDriver with "Unable to match sources". Their data IDs are: --id visit=34338 ccd=75 --id visit=34362 ccd=38 --id visit=34382 ccd=43 --id visit=34384 ccd=88 --id visit=34402 ccd=61 --id visit=34422 ccd=42 --id visit=34424 ccd=2 --id visit=34448 ccd=79 --id visit=34450 ccd=25 --id visit=34478 ccd=21 --id visit=34482 ccd=50 --id visit=34644 ccd=72 --id visit=34690 ccd=39 --id visit=36140 ccd=30 --id visit=36170 ccd=18 --id visit=36182 ccd=80 --id visit=36212 ccd=34 --id visit=36234 ccd=82 --id visit=36240 ccd=0 --id visit=36258 ccd=73 --id visit=36428 ccd=61 --id visit=36432 ccd=23 --id visit=36492 ccd=50 --id visit=34942 ccd=44 --id visit=36762 ccd=18 --id visit=36774 ccd=24 --id visit=36792 ccd=34 --id visit=36808 ccd=26 --id visit=36828 ccd=16 --id visit=36828 ccd=73 --id visit=26046 ccd=32 --id visit=26050 ccd=74 --id visit=26058 ccd=34 --id visit=26060 ccd=70 -id visit=26072 ccd=12 --id visit=26080 ccd=66 --id visit=26084 ccd=64 --id visit=23864 ccd=12 --id visit=1308 ccd=40 --id visit=23224 ccd=59 --id visit=23232 ccd=32 --id visit=27106 ccd=33 --id visit=27106 ccd=41 --id visit=27128 ccd=26 --id visit=27134 ccd=28 --id visit=440 ccd=92 --id visit=452 ccd=35 --id visit=452 ccd=42 --id visit=472 ccd=30 --id visit=1246 ccd=19 --id visit=19696 ccd=20 --id visit=30500 ccd=20 --id visit=1180 ccd=89 --id visit=1184 ccd=26 --id visit=1194 ccd=26 --id visit=360 ccd=26 --id visit=22628 ccd=18 --id visit=22664 ccd=26 --id visit=23046 ccd=26 --id visit=23050 ccd=84 This looks related to the astrometry matcher default change to PessimisticB by DM-14857 , which got merged before w_2019_04. DM-17843 has been filed in case further investigation is needed. There is a jump in many metrics shown in squash.
            Hide
            Permalink
            hchiang2 Hsin-Fang Chiang added a comment -

            c.f. processCcd failures before the matcher change, there were only 9 CCDs failed in processCcd (e.g. DM-16110)

            Show
            hchiang2 Hsin-Fang Chiang added a comment - c.f. processCcd failures before the matcher change, there were only 9 CCDs failed in processCcd (e.g. DM-16110 )
            Hide
            Permalink
            hchiang2 Hsin-Fang Chiang added a comment -

            The first attempt of running multiband for the COSMOS tract failed with file IO issues, likely related to the /lsst disk full on the same night (Feb12-13) as reported in Slack #dm-infrastructure.

            coadd/multiband: the number of data products and empty patches are the same as w_2018_15/DM-14123 (this comment)

            Show
            hchiang2 Hsin-Fang Chiang added a comment - The first attempt of running multiband for the COSMOS tract failed with file IO issues, likely related to the /lsst disk full on the same night (Feb12-13) as reported in Slack #dm-infrastructure. coadd/multiband: the number of data products and empty patches are the same as w_2018_15/ DM-14123 ( this comment )
            Hide
            Permalink
            hchiang2 Hsin-Fang Chiang added a comment -

            The logs have been copied to /datasets/hsc/repo/rerun/RC/w_2019_06/DM-17400/logs/

            The pipe_analysis and validate_drp runs are at DM-17401 . The jointcal processing is at DM-17402.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - The logs have been copied to /datasets/hsc/repo/rerun/RC/w_2019_06/ DM-17400 /logs/ The pipe_analysis and validate_drp runs are at DM-17401 . The jointcal processing is at DM-17402 .

              People

              Assignee:
              hchiang2 Hsin-Fang Chiang
              Reporter:
              hchiang2 Hsin-Fang Chiang
              Watchers:
              Hsin-Fang Chiang
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.