Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-14123

Reprocess RC2 with w_2018_15

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Reprocess the HSC RC2 dataset, as defined in DM-11345, using Stack w_2018_15. Steps include makeSkyMap.py, singleFrameDriver.py, mosaic.py, skyCorrection.py, coaddDriver.py, and multiBandDriver.py. 

      (Starting this week, the reprocessing actually has RC2 only, and no longer include the two WIDE tracts of RC1.)

        Attachments

          Issue Links

          relates to

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-13665 Finalize the stack version, step, and config for the S18 PDR1 reprocessing

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-14137 Run pipe_analysis scripts with the RC2 data of w_2018_15

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-14144 Run validate_drp with w_2018_15 HSC RC2 outputs

          • Done

            Activity

            Ascending order - Click to sort in descending order
            No builds found.
            hchiang2 Hsin-Fang Chiang created issue -
            hchiang2 Hsin-Fang Chiang made changes -
            Field Original Value New Value
            Epic Link DM-13926 [ 39699 ]
            hchiang2 Hsin-Fang Chiang made changes -
            Link This issue relates to DM-13665 [ DM-13665 ]
            hchiang2 Hsin-Fang Chiang made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            Hide
            Permalink
            hchiang2 Hsin-Fang Chiang added a comment -

            For the 432 visits of the RC2 dataset, 7 CCDs failed processCcd/singleFrameDriver, including 1 failing with "RuntimeError: Unable to match sources"

            --id visit=34412 ccd=78

            and 6 with "RuntimeError: No matches to use for photocal"

            --id visit=35892 ccd=12
            		 
            --id visit=36216 ccd=6
            		 
            --id visit=27068 ccd=96
            		 
            --id visit=19712 ccd=33
            		 
            --id visit=17934 ccd=1
            		 
            --id visit=23596 ccd=6

             

            This is the same since w_2018_10/DM-13647.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - For the 432 visits of the RC2 dataset, 7 CCDs failed processCcd/singleFrameDriver, including 1 failing with "RuntimeError: Unable to match sources" --id visit=34412 ccd=78 and 6 with "RuntimeError: No matches to use for photocal" --id visit=35892 ccd=12 --id visit=36216 ccd=6 --id visit=27068 ccd=96 --id visit=19712 ccd=33 --id visit=17934 ccd=1 --id visit=23596 ccd=6   This is the same since w_2018_10/ DM-13647 .
            Hide
            Permalink
            hchiang2 Hsin-Fang Chiang added a comment -

            The error of DM-13517 (IndexError: cannot do a non-empty take from an empty axes) has been fixed and no longer seen in the RC2 dataset.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - The error of DM-13517 ( IndexError: cannot do a non-empty take from an empty axes ) has been fixed and no longer seen in the RC2 dataset.
            hchiang2 Hsin-Fang Chiang made changes -
            Link This issue relates to DM-14137 [ DM-14137 ]
            hchiang2 Hsin-Fang Chiang made changes -
            Link This issue relates to DM-14144 [ DM-14144 ]
            Hide
            Permalink
            hchiang2 Hsin-Fang Chiang added a comment -

            For the COSMOS tract 9813, the coadd products have 78 patches in HSC-G, 79 in HSC-I, 74 in HSC-R, 79 in HSC-Y, 78 in HSC-Z, and 76 in NB0921; in total 464 patches. This is one more patch than the w_2018_14 run (DM-13890) at HSC-Z '0,7', as DM-13517 has been fixed.

            The empty patches of tract 9813 are:

            • patch 0,0 HSC-G,HSC-I,HSC-R,HSC-Y,HSC-Z,NB0921
            • patch 0,1 HSC-R
            • patch 0,7 HSC-R,NB0921
            • patch 0,8 HSC-G,HSC-I,HSC-R,HSC-Y,HSC-Z,NB0921
            • patch 1,8 HSC-R,NB0921
            • patch 8,0 HSC-G,HSC-R,HSC-Z
            • patch 8,8 HSC-R,NB0921

            In the two WIDE tracts 9697 and 9615, each has all 81 patches of coadd products in all 5 bands.

            The MultiBand products have 81 patches in both WIDE tracts, 79 patches in COSMOS tract 9813.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - For the COSMOS tract 9813, the coadd products have 78 patches in HSC-G, 79 in HSC-I, 74 in HSC-R, 79 in HSC-Y, 78 in HSC-Z, and 76 in NB0921; in total 464 patches. This is one more patch than the w_2018_14 run ( DM-13890 ) at HSC-Z '0,7', as DM-13517 has been fixed. The empty patches of tract 9813 are: patch 0,0 HSC-G,HSC-I,HSC-R,HSC-Y,HSC-Z,NB0921 patch 0,1 HSC-R patch 0,7 HSC-R,NB0921 patch 0,8 HSC-G,HSC-I,HSC-R,HSC-Y,HSC-Z,NB0921 patch 1,8 HSC-R,NB0921 patch 8,0 HSC-G,HSC-R,HSC-Z patch 8,8 HSC-R,NB0921 In the two WIDE tracts 9697 and 9615, each has all 81 patches of coadd products in all 5 bands. The MultiBand products have 81 patches in both WIDE tracts, 79 patches in COSMOS tract 9813.
            Hide
            Permalink
            hchiang2 Hsin-Fang Chiang added a comment -

            The output repo is at /datasets/hsc/repo/rerun/RC/w_2018_15/DM-14123/
            The logs have been copied to /datasets/hsc/repo/rerun/RC/w_2018_15/DM-14123/logs/
            The pipe_analysis run is in DM-14137.
            The validate_drp run is in DM-14144.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - The output repo is at /datasets/hsc/repo/rerun/RC/w_2018_15/ DM-14123 / The logs have been copied to /datasets/hsc/repo/rerun/RC/w_2018_15/ DM-14123 /logs/ The pipe_analysis run is in DM-14137 . The validate_drp run is in DM-14144 .
            hchiang2 Hsin-Fang Chiang made changes -
            Description Reprocess the HSC RC2 dataset, as defined in DM-11345, using Stack w_2018_15. Steps include {{makeSkyMap.py}}, {{singleFrameDriver.py}}, {{mosaic.py}}, {{skyCorrection.py}}, {{coaddDriver.py}}, and {{multiBandDriver.py}}.  Reprocess the HSC RC2 dataset, as defined in DM-11345, using Stack w_2018_15. Steps include {{makeSkyMap.py}}, {{singleFrameDriver.py}}, {{mosaic.py}}, {{skyCorrection.py}}, {{coaddDriver.py}}, and {{multiBandDriver.py}}. 

            (Starting this week, the reprocessing actually has RC2 only, and no longer include the two WIDE tracts of RC1.)
            hchiang2 Hsin-Fang Chiang made changes -
            Resolution Done [ 10000 ]
            Status In Progress [ 3 ] Done [ 10002 ]
            hchiang2 Hsin-Fang Chiang made changes -
            Story Points 3
            hchiang2 Hsin-Fang Chiang made changes -
            Risk Score 0
            hchiang2 Hsin-Fang Chiang made changes -
            Comment [ coadd/multiband: the number of data products and empty patches are the same as w_2018_15/DM-14123 ([this comment|https://jira.lsstcorp.org/browse/DM-14123?focusedCommentId=100481&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-100481]) ]

              People

              Assignee:
              hchiang2 Hsin-Fang Chiang
              Reporter:
              hchiang2 Hsin-Fang Chiang
              Watchers:
              Hsin-Fang Chiang
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.