Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-14990

Run pipe_analysis and validate_drp with w_2018_30 HSC RC2 outputs

    XMLWordPrintable

Details

    • Story
    • Status: Done
    • Resolution: Done
    • None
    • None
    • None

    Description

      Run pipe_analysis scripts

      1. visitAnalysis
      2. coaddAnalysis
      3. colorAnalysis
      4. compareVisitAnalysis
      5. compareCoaddAnalysis

      and validate_drp

      1. matchedVisitMetrics.py
      2. validateDrp.py
      3. reportPerformance.py

      Upload the validate_drp metrics (json files) to SQuaSH:

      1. dispatch_verify.py

      with the w_2018_30 RC2 outputs from DM-14988

      Attachments

        Issue Links

          Activity

            The version of pipe_analysis was master at the time of the w_2018_30 release, at commit 2f72a4e.
            The comparison scripts were run against RC/w_2018_28/DM-14988

            hchiang2 Hsin-Fang Chiang added a comment - The version of pipe_analysis was master at the time of the w_2018_30 release, at commit 2f72a4e . The comparison scripts were run against RC/w_2018_28/ DM-14988

            Like before, filter=NB0921 is ignored for validateDrp.py due to DM-15127.

            hchiang2 Hsin-Fang Chiang added a comment - Like before, filter=NB0921 is ignored for validateDrp.py due to DM-15127 .

            pipe_analysis logs are at
            /datasets/hsc/repo/rerun/RC/w_2018_30/DM-15120/qaLogs/

            validate_drp results and logs are at
            /datasets/hsc/repo/rerun/RC/w_2018_30/DM-15120/validateDrp/

            They can also be accessed via
            https://lsst-web.ncsa.illinois.edu/~hchiang2/RC2_w_2018_30/

             

            hchiang2 Hsin-Fang Chiang added a comment - pipe_analysis logs are at /datasets/hsc/repo/rerun/RC/w_2018_30/ DM-15120 /qaLogs/ validate_drp results and logs are at /datasets/hsc/repo/rerun/RC/w_2018_30/ DM-15120 /validateDrp/ They can also be accessed via https://lsst-web.ncsa.illinois.edu/~hchiang2/RC2_w_2018_30/  

            validate_drp results of all 6 filters in the COSMOS tract were uploaded to SQuaSH (dataset = HSC RC2) at https://squash.lsst.codes/ on Aug 3.

            To do so using dispatch_verify.py, user environment variables need to be set (e.g. notes on https://confluence.lsstcorp.org/display/~hchiang2/validateDrp#validateDrp-Sendverify.JobJSONfilestoSQuaSH )
            I mistakenly used the old and wrong setup (VERSION_TAG=w_2018_28, RUN_ID, RUN_ID_URL), rather than the w_2018_30 ticket. However, they don't seem to matter as displayed on https://squash.lsst.codes/dash/code_changes/ (those detailed information seem not shown at this moment) so I'm not re-uploading for now.

            hchiang2 Hsin-Fang Chiang added a comment - validate_drp results of all 6 filters in the COSMOS tract were uploaded to SQuaSH (dataset = HSC RC2) at  https://squash.lsst.codes/ on Aug 3. To do so using dispatch_verify.py , user environment variables need to be set (e.g. notes on https://confluence.lsstcorp.org/display/~hchiang2/validateDrp#validateDrp-Sendverify.JobJSONfilestoSQuaSH ) I mistakenly used the old and wrong setup (VERSION_TAG=w_2018_28, RUN_ID, RUN_ID_URL), rather than the w_2018_30 ticket. However, they don't seem to matter as displayed on https://squash.lsst.codes/dash/code_changes/ (those detailed information seem not shown at this moment) so I'm not re-uploading for now.

            From Slack conversations, afausti pointed out that the information is stored in the SQuaSH database as environment metadata.  

            Currently, unlike afausti, I do not have the admin access to the SQuaSH database for correcting the records. When the time comes we will discuss database access and related topics. 

            hchiang2 Hsin-Fang Chiang added a comment - From Slack conversations, afausti pointed out that the information is stored in the SQuaSH database as environment metadata.   Currently, unlike afausti , I do not have the admin access to the SQuaSH database for correcting the records. When the time comes we will discuss database access and related topics. 

            afausti has kindly corrected the records in the SQuaSH database in DM-15407.

            hchiang2 Hsin-Fang Chiang added a comment - afausti has kindly corrected the records in the SQuaSH database in DM-15407 .

            People

              hchiang2 Hsin-Fang Chiang
              hchiang2 Hsin-Fang Chiang
              Hsin-Fang Chiang
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.