Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-19128

Run pipe_analysis & validate_drp with w_2019_14 HSC RC2

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Run pipe_analysis scripts

      1. visitAnalysis
      2. coaddAnalysis
      3. colorAnalysis
      4. compareVisitAnalysis
      5. compareCoaddAnalysis

      and validate_drp

      1. matchedVisitMetrics.py
      2. validateDrp.py
      3. reportPerformance.py

      Upload the validate_drp metrics (json files) to SQuaSH:

      1. dispatch_verify.py

      with the w_2019_14 RC2 outputs from DM-18300

        Attachments

          Issue Links

            Activity

            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            Use w_2019_14 and the master branch of pipe_analysis at the time of w_2019_14 release (at commit a574165).

            The pipe_analysis comparison scripts are run against the rerun of RC/w_2019_10/DM-17940

            Since last month for colorAnalysis, also set up lsst_sims to match the weekly lsst_distrib; the meta-package with the sims_w_2019_14 tag is used. This is to get the per-source Galactic Extinction applied.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - Use  w_2019_14 and the master branch of pipe_analysis at the time of w_2019_14 release (at commit a574165 ). The  pipe_analysis comparison scripts are run against the rerun of RC/w_2019_10/ DM-17940 Since last month for colorAnalysis, also set up lsst_sims to match the weekly lsst_distrib ; the meta-package with the sims_w_2019_14 tag is used. This is to get the per-source Galactic Extinction applied.
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            With w_2019_14, matchedVisitMetrics.py, validateDrp.py and reportPerformance.py were run with SFM and mosaic results. filter=NB0921 is ignored for validateDrp.py due to DM-15127.

            validate_drp results and logs are at
            /datasets/hsc/repo/rerun/RC/w_2019_14/DM-18300/validateDrp/
            and the results of all 6 filters in the COSMOS tract are uploaded to SQuaSH (dataset = HSC RC2) at https://squash.lsst.codes/

            Show
            hchiang2 Hsin-Fang Chiang added a comment - With w_2019_14 , matchedVisitMetrics.py , validateDrp.py and reportPerformance.py were run with SFM and mosaic results. filter=NB0921 is ignored for validateDrp.py due to DM-15127 . validate_drp results and logs are at /datasets/hsc/repo/rerun/RC/w_2019_14/ DM-18300 /validateDrp/ and the results of all 6 filters in the COSMOS tract are uploaded to SQuaSH (dataset = HSC RC2) at https://squash.lsst.codes/
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            Some metrics shown in SQuaSH have increased; this is expected by the AM1 computation changes in DM-18638

            Show
            hchiang2 Hsin-Fang Chiang added a comment - Some metrics shown in SQuaSH have increased; this is expected by the AM1 computation changes in DM-18638
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            compareVisitAnalysis and compareCoaddAnalysis failed because pipe_analysis wasn't updated for the DM-10156 changes; DM-19185 has been filed.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - compareVisitAnalysis and compareCoaddAnalysis failed because pipe_analysis wasn't updated for the DM-10156 changes; DM-19185 has been filed.
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            compareVisitAnalysis and compareCoaddAnalysis were re-done using pipe_analysis after the DM-19185 fix was merged.

            coaddAnalysis.py, compareCoaddAnalysis.py, and colorAnalysis.py used the rerun repo of RC/w_2019_14/DM-18300-jointcal

            Show
            hchiang2 Hsin-Fang Chiang added a comment - compareVisitAnalysis and compareCoaddAnalysis were re-done using pipe_analysis after the DM-19185 fix was merged. coaddAnalysis.py, compareCoaddAnalysis.py, and colorAnalysis.py used the rerun repo of RC/w_2019_14/ DM-18300 -jointcal
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            pipe_analysis logs are at
            /datasets/hsc/repo/rerun/RC/w_2019_14/DM-18300-jointcal/qaLogs/

             

            They can also be accessed via 
            https://lsst-web.ncsa.illinois.edu/~hchiang2/RC2_w_2019_14

            Show
            hchiang2 Hsin-Fang Chiang added a comment - pipe_analysis logs are at /datasets/hsc/repo/rerun/RC/w_2019_14/ DM-18300 -jointcal/qaLogs/   They can also be accessed via  https://lsst-web.ncsa.illinois.edu/~hchiang2/RC2_w_2019_14
            Hide
            lauren Lauren MacArthur added a comment -

            It looks like the above link points to the wrong rerun (it leads to https://lsst-web.ncsa.illinois.edu/~hchiang2/RC2_w_2019_06).

            Show
            lauren Lauren MacArthur added a comment - It looks like the above link points to the wrong rerun (it leads to  https://lsst-web.ncsa.illinois.edu/~hchiang2/RC2_w_2019_06 ).
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            Thanks Lauren for pointing it out!

            Somehow the link wasn't set correctly. It should be
            https://lsst-web.ncsa.illinois.edu/~hchiang2/RC2_w_2019_14

            Show
            hchiang2 Hsin-Fang Chiang added a comment - Thanks Lauren for pointing it out! Somehow the link wasn't set correctly. It should be https://lsst-web.ncsa.illinois.edu/~hchiang2/RC2_w_2019_14

              People

              • Assignee:
                hchiang2 Hsin-Fang Chiang
                Reporter:
                hchiang2 Hsin-Fang Chiang
                Watchers:
                Hsin-Fang Chiang, Lauren MacArthur
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel