Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-14201

coaddDriver of the HSC PDR1 dataset

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      coaddDriver.py of the entire HSC PDR1 dataset with w_2018_15

        Attachments

          Issue Links

            Activity

            No builds found.
            hchiang2 Hsin-Fang Chiang created issue -
            hchiang2 Hsin-Fang Chiang made changes -
            Field Original Value New Value
            Epic Link DM-13926 [ 39699 ]
            hchiang2 Hsin-Fang Chiang made changes -
            Link This issue is child task of DM-13666 [ DM-13666 ]
            hchiang2 Hsin-Fang Chiang made changes -
            Risk Score 0
            hchiang2 Hsin-Fang Chiang made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            The execution unit of coaddDriver was the same as mosaic (DM-14200). It was run for each tract x filter combination, using all visits overlapping that tract in that filter. Each layer was done separately. There were 69 slurm jobs in UDEEP, 218 in DEEP, and 455 in WIDE (same as in DM-14200).

            Show
            hchiang2 Hsin-Fang Chiang added a comment - The execution unit of coaddDriver was the same as mosaic ( DM-14200 ). It was run for each tract x filter combination, using all visits overlapping that tract in that filter. Each layer was done separately. There were 69 slurm jobs in UDEEP, 218 in DEEP, and 455 in WIDE (same as in DM-14200 ).
            hchiang2 Hsin-Fang Chiang made changes -
            Link This issue relates to DM-14282 [ DM-14282 ]
            hchiang2 Hsin-Fang Chiang made changes -
            Link This issue relates to DM-14286 [ DM-14286 ]
            hchiang2 Hsin-Fang Chiang made changes -
            Link This issue relates to DM-14303 [ DM-14303 ]
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            The 1484 log files from the 742 jobs have been copied to

            /datasets/hsc/repo/rerun/DM-13666/UDEEP/logs/coadd/
            /datasets/hsc/repo/rerun/DM-13666/DEEP/logs/coadd/
            /datasets/hsc/repo/rerun/DM-13666/WIDE/logs/coadd/
            

            There are no FATAL- or ERROR-level log records found in the log files. Many WARN-level records appeared. I've filed 3 new tickets for some new non-fatal errors I noticed and the Science Pipeline team might wish to investigate more and fix. I think all cases are edge patches with little coverage, but did not check all images one by one. The three tickets are:

            • DM-14282: IndexError: cannot do a non-empty take from an empty axes. at detectCoaddSources.runDetection in 8 patches, 2 in UDEEP, 5 in DEEP, and 1 in WIDE.
            • DM-14286: RuntimeError: No PsfMatched warps were found to build the template coadd which is required to run CompareWarpAssembleCoaddTask. while there are 0 good pixels for psfMatchedWarp so no psfMatchedWarp files were written. These happened in 5 patches, 4 in DEEP and 1 in WIDE.
            • DM-14303: RuntimeError: Variance rescaling factor (20.491061) exceeds configured limit (10.000000) in one edge patch in WIDE.
            Show
            hchiang2 Hsin-Fang Chiang added a comment - The 1484 log files from the 742 jobs have been copied to /datasets/hsc/repo/rerun/DM-13666/UDEEP/logs/coadd/ /datasets/hsc/repo/rerun/DM-13666/DEEP/logs/coadd/ /datasets/hsc/repo/rerun/DM-13666/WIDE/logs/coadd/ There are no FATAL- or ERROR-level log records found in the log files. Many WARN-level records appeared. I've filed 3 new tickets for some new non-fatal errors I noticed and the Science Pipeline team might wish to investigate more and fix. I think all cases are edge patches with little coverage, but did not check all images one by one. The three tickets are: DM-14282 : IndexError: cannot do a non-empty take from an empty axes. at detectCoaddSources.runDetection in 8 patches, 2 in UDEEP, 5 in DEEP, and 1 in WIDE. DM-14286 : RuntimeError: No PsfMatched warps were found to build the template coadd which is required to run CompareWarpAssembleCoaddTask. while there are 0 good pixels for psfMatchedWarp so no psfMatchedWarp files were written. These happened in 5 patches, 4 in DEEP and 1 in WIDE. DM-14303 : RuntimeError: Variance rescaling factor (20.491061) exceeds configured limit (10.000000) in one edge patch in WIDE.
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            The data IDs to reproduce the above three cases can be found in the tickets. 

            Show
            hchiang2 Hsin-Fang Chiang added a comment - The data IDs to reproduce the above three cases can be found in the tickets. 
            hchiang2 Hsin-Fang Chiang made changes -
            Story Points 4
            hchiang2 Hsin-Fang Chiang made changes -
            Resolution Done [ 10000 ]
            Status In Progress [ 3 ] Done [ 10002 ]

              People

              Assignee:
              hchiang2 Hsin-Fang Chiang
              Reporter:
              hchiang2 Hsin-Fang Chiang
              Watchers:
              Hsin-Fang Chiang
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.