Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-23807

coaddAnalysis for HSC-PDR2

    XMLWordPrintable

Details

    • Story
    • Status: Done
    • Resolution: Done
    • None
    • None
    • None

    Description

      Use w_2020_08 stack with qa_explorer at commit ab69304 and pipe_analysis at commit 09a7675. (same as DM-23579)

      The command line will look like this, with all tract/filter IDs pairs, with either DEEP or WIDE accordingly

      coaddAnalysis.py /datasets/hsc/repo/ --calib /datasets/hsc/calib/20200115/ --rerun DM-23243/MULTIBAND/DEEP:DM-23243/ANALYSIS/DEEP --doraise --config doWriteParquetTables=True --id tract=08766 filter=NB0921
      

      Attachments

        Issue Links

          Activity

            (Uploading the data IDs of the 36 UnboundLocalError in DEEP+UDEEP)

            hchiang2 Hsin-Fang Chiang added a comment - (Uploading the data IDs of the 36 UnboundLocalError in DEEP+UDEEP)

            For WIDE, I got 3319 jobs in total; 3319 log files are at
            /datasets/hsc/repo/rerun/DM-23243/logs/coaddAnalysis/wide/

            There are

            • 1429 UnboundLocalError: local variable 'axes2' referenced before assignment
            • 11 RuntimeError: No good data points to plot for sample labelled: star

            Nonetheless, there are 3308 sets of parquet outputs:
            /datasets/hsc/repo/rerun/DM-23243/ANALYSIS/WIDE/plots/*/*/*_forced.parq
            /datasets/hsc/repo/rerun/DM-23243/ANALYSIS/WIDE/plots/*/*/*_unforced.parq
            but more missing plots.

            hchiang2 Hsin-Fang Chiang added a comment - For WIDE, I got 3319 jobs in total; 3319 log files are at /datasets/hsc/repo/rerun/ DM-23243 /logs/coaddAnalysis/wide/ There are 1429 UnboundLocalError: local variable 'axes2' referenced before assignment 11 RuntimeError: No good data points to plot for sample labelled: star Nonetheless, there are 3308 sets of parquet outputs: /datasets/hsc/repo/rerun/ DM-23243 /ANALYSIS/WIDE/plots/*/*/*_forced.parq /datasets/hsc/repo/rerun/ DM-23243 /ANALYSIS/WIDE/plots/*/*/*_unforced.parq but more missing plots.

            (Uploading the data IDs of the two kinds of errors for WIDE)

            hchiang2 Hsin-Fang Chiang added a comment - (Uploading the data IDs of the two kinds of errors for WIDE)

            Note that some of these jobs seem memory demanding. For WIDE, the high observed was ~75GB.

            hchiang2 Hsin-Fang Chiang added a comment - Note that some of these jobs seem memory demanding. For WIDE, the high observed was ~75GB.

            Slurm job IDs:

            • DEEP: 245058,245693,245716
            • WIDE: 246777,246779,246783,246785,246798,246799,246800

            Commands of the jobs can be found at:
            https://github.com/lsst-dm/s20-hsc-pdr2-reprocessing/tree/master/11_coaddAnalysis

            I'm closing this done. See above comments for logs and noted errors. The generated parquet files allowed the subsequent task qa_explorer/matchVisits.py to be run (DM-23831). If we want a re-run post-DM-23829, please file a new ticket.

            hchiang2 Hsin-Fang Chiang added a comment - Slurm job IDs: DEEP: 245058,245693,245716 WIDE: 246777,246779,246783,246785,246798,246799,246800 Commands of the jobs can be found at: https://github.com/lsst-dm/s20-hsc-pdr2-reprocessing/tree/master/11_coaddAnalysis I'm closing this done. See above comments for logs and noted errors. The generated parquet files allowed the subsequent task qa_explorer/matchVisits.py to be run ( DM-23831 ). If we want a re-run post- DM-23829 , please file a new ticket.

            People

              hchiang2 Hsin-Fang Chiang
              hchiang2 Hsin-Fang Chiang
              Hsin-Fang Chiang, Lauren MacArthur
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.