Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-14201

coaddDriver of the HSC PDR1 dataset

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      coaddDriver.py of the entire HSC PDR1 dataset with w_2018_15

        Attachments

          Issue Links

          is child task of

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-13666 S18 reprocessing of the HSC SSP PDR1 dataset

          • Done
          relates to

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-14286 CompareWarpAssembleCoadd RuntimeError when No PsfMatched warps found at edge cases

          • To Do

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-14282 IndexError in detectCoaddSources scaleVariance

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-14303 detectCoaddSources RuntimeError: Variance rescaling factor exceeds configured limit

          • Won't Fix

            Activity

            Ascending order - Click to sort in descending order
            Hide
            Permalink
            hchiang2 Hsin-Fang Chiang added a comment -

            The execution unit of coaddDriver was the same as mosaic (DM-14200). It was run for each tract x filter combination, using all visits overlapping that tract in that filter. Each layer was done separately. There were 69 slurm jobs in UDEEP, 218 in DEEP, and 455 in WIDE (same as in DM-14200).

            Show
            hchiang2 Hsin-Fang Chiang added a comment - The execution unit of coaddDriver was the same as mosaic ( DM-14200 ). It was run for each tract x filter combination, using all visits overlapping that tract in that filter. Each layer was done separately. There were 69 slurm jobs in UDEEP, 218 in DEEP, and 455 in WIDE (same as in DM-14200 ).
            Hide
            Permalink
            hchiang2 Hsin-Fang Chiang added a comment -

            The 1484 log files from the 742 jobs have been copied to

            /datasets/hsc/repo/rerun/DM-13666/UDEEP/logs/coadd/
            		 
            /datasets/hsc/repo/rerun/DM-13666/DEEP/logs/coadd/
            		 
            /datasets/hsc/repo/rerun/DM-13666/WIDE/logs/coadd/

            There are no FATAL- or ERROR-level log records found in the log files. Many WARN-level records appeared. I've filed 3 new tickets for some new non-fatal errors I noticed and the Science Pipeline team might wish to investigate more and fix. I think all cases are edge patches with little coverage, but did not check all images one by one. The three tickets are:

            • DM-14282: IndexError: cannot do a non-empty take from an empty axes. at detectCoaddSources.runDetection in 8 patches, 2 in UDEEP, 5 in DEEP, and 1 in WIDE.
            • DM-14286: RuntimeError: No PsfMatched warps were found to build the template coadd which is required to run CompareWarpAssembleCoaddTask. while there are 0 good pixels for psfMatchedWarp so no psfMatchedWarp files were written. These happened in 5 patches, 4 in DEEP and 1 in WIDE.
            • DM-14303: RuntimeError: Variance rescaling factor (20.491061) exceeds configured limit (10.000000) in one edge patch in WIDE.
            Show
            hchiang2 Hsin-Fang Chiang added a comment - The 1484 log files from the 742 jobs have been copied to /datasets/hsc/repo/rerun/DM-13666/UDEEP/logs/coadd/ /datasets/hsc/repo/rerun/DM-13666/DEEP/logs/coadd/ /datasets/hsc/repo/rerun/DM-13666/WIDE/logs/coadd/ There are no FATAL- or ERROR-level log records found in the log files. Many WARN-level records appeared. I've filed 3 new tickets for some new non-fatal errors I noticed and the Science Pipeline team might wish to investigate more and fix. I think all cases are edge patches with little coverage, but did not check all images one by one. The three tickets are: DM-14282 : IndexError: cannot do a non-empty take from an empty axes. at detectCoaddSources.runDetection in 8 patches, 2 in UDEEP, 5 in DEEP, and 1 in WIDE. DM-14286 : RuntimeError: No PsfMatched warps were found to build the template coadd which is required to run CompareWarpAssembleCoaddTask. while there are 0 good pixels for psfMatchedWarp so no psfMatchedWarp files were written. These happened in 5 patches, 4 in DEEP and 1 in WIDE. DM-14303 : RuntimeError: Variance rescaling factor (20.491061) exceeds configured limit (10.000000) in one edge patch in WIDE.
            Hide
            Permalink
            hchiang2 Hsin-Fang Chiang added a comment -

            The data IDs to reproduce the above three cases can be found in the tickets. 

            Show
            hchiang2 Hsin-Fang Chiang added a comment - The data IDs to reproduce the above three cases can be found in the tickets. 

              People

              Assignee:
              hchiang2 Hsin-Fang Chiang
              Reporter:
              hchiang2 Hsin-Fang Chiang
              Watchers:
              Hsin-Fang Chiang
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.