Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-13804

Run validate_drp with w_2018_10 HSC RC2 outputs

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Run matchedVisitMetrics.py and validateDrp.py with the HSC RC2 w_2018_10 outputs from DM-13647.

      Follow Michael Wood-Vasey's example:

      # Need to specify tract, otherwise one will get a meas_base failure.
      matchedVisitMetrics.py "${REPO}" --output "${OUTREPO}" --config useJointCal=True --id visit=${VISITIDS} tract=${TRACT}
       
      # Capture the output
      validateDrp.py "${OUTREPO}"/matchedVisit_${FILTER}.json --noplot > ${OUTREPO}/metrics.log
      

      Use the ticket branch of DM-13742.

        Attachments

          Issue Links

            Activity

            No builds found.
            hchiang2 Hsin-Fang Chiang created issue -
            hchiang2 Hsin-Fang Chiang made changes -
            Field Original Value New Value
            Epic Link DM-12831 [ 36488 ]
            hchiang2 Hsin-Fang Chiang made changes -
            Link This issue relates to DM-13647 [ DM-13647 ]
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            The HSC RC2 dataset includes data in filter=NB0921 in the COSMOS tract; it will be ignored here because filter metric definitions need to be added.

              File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_base/14.0-6-ge2c9487+51/python/lsst/pipe/base/cmdLineTask.py", line 586, in parseAndRun
                resultList = taskRunner.run(parsedCmd)
              File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_base/14.0-6-ge2c9487+51/python/lsst/pipe/base/cmdLineTask.py", line 240, in run
                resultList = list(mapFunc(self, targetList))
              File "/home/hchiang2/stack/validate_drp/python/lsst/validate/drp/matchedVisitMetricsTask.py", line 36, in __call__
                return task.run(*args)
              File "/home/hchiang2/stack/validate_drp/python/lsst/validate/drp/matchedVisitMetricsTask.py", line 133, in run
                verbose=self.config.verbose)
              File "/home/hchiang2/stack/validate_drp/python/lsst/validate/drp/validate.py", line 311, in runOneFilter
                job=job, linkedBlobs=linkedBlobs)
              File "/home/hchiang2/stack/validate_drp/python/lsst/validate/drp/calcsrd/pf1.py", line 72, in __init__
                PA2.get_spec(spec_name, filter_name=self.filter_name)
              File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/validate_base/master-g24db930f3a+22/python/lsst/validate/base/metric.py", line 381, in get_spec
                name, filter_name, self.name))
            RuntimeError: No PA2 spec found for name=stretch filter_name=NB0921
            

            Show
            hchiang2 Hsin-Fang Chiang added a comment - The HSC RC2 dataset includes data in filter=NB0921 in the COSMOS tract; it will be ignored here because filter metric definitions need to be added. File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_base/14.0-6-ge2c9487+51/python/lsst/pipe/base/cmdLineTask.py", line 586, in parseAndRun resultList = taskRunner.run(parsedCmd) File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_base/14.0-6-ge2c9487+51/python/lsst/pipe/base/cmdLineTask.py", line 240, in run resultList = list(mapFunc(self, targetList)) File "/home/hchiang2/stack/validate_drp/python/lsst/validate/drp/matchedVisitMetricsTask.py", line 36, in __call__ return task.run(*args) File "/home/hchiang2/stack/validate_drp/python/lsst/validate/drp/matchedVisitMetricsTask.py", line 133, in run verbose=self.config.verbose) File "/home/hchiang2/stack/validate_drp/python/lsst/validate/drp/validate.py", line 311, in runOneFilter job=job, linkedBlobs=linkedBlobs) File "/home/hchiang2/stack/validate_drp/python/lsst/validate/drp/calcsrd/pf1.py", line 72, in __init__ PA2.get_spec(spec_name, filter_name=self.filter_name) File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/validate_base/master-g24db930f3a+22/python/lsst/validate/base/metric.py", line 381, in get_spec name, filter_name, self.name)) RuntimeError: No PA2 spec found for name=stretch filter_name=NB0921
            hchiang2 Hsin-Fang Chiang made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            Hide
            wmwood-vasey Michael Wood-Vasey added a comment -

            A final command to add once the JSON output files are generated would be reportPerformance.py. This produces a text file report_performance.rst summarizing the KPMs for each filter. E.g., if the outputs are in JSON files in directories such as w_2018_08_HSC-R/ and w_2018_08_HSC-I, then

            reportPerformance.py w_2018_08_HSC-*/*.json
            

            will create a report_performance.rst file with the following table summary:

            [wmwv@lsst-dev01 HSC_RC_analysis]$ cat report_performance.rst
            ====== ====== ======= ======== ======================= ======== ========
            Metric Filter    Unit Operator SRD Requirement: design    Value Comments
            ====== ====== ======= ======== ======================= ======== ========
               AM1  HSC-I marcsec       <=                    10.0     4.63
               AF1  HSC-I               <=                    10.0      4.4
               AD1  HSC-I marcsec       <=                    20.0     14.4
               AM2  HSC-I marcsec       <=                    10.0     4.12
               AF2  HSC-I               <=                    10.0     4.45
               AD2  HSC-I marcsec       <=                    20.0     14.5
               AM3  HSC-I marcsec       <=                    15.0        0
               AF3  HSC-I               <=                    10.0     0.83
               AD3  HSC-I marcsec       <=                    30.0     12.1
               PA1  HSC-I    mmag       <=                     5.0       44
               PA2  HSC-I    mmag       <=                    15.0      320
               PF1  HSC-I               <=                    10.0     68.9
               TE1  HSC-I               <=                   2e-05 3.57e-06
               TE2  HSC-I               <=                   5e-05 3.71e-07
               AM1  HSC-R marcsec       <=                    10.0        0
               AF1  HSC-R               <=                    10.0     3.25
               AD1  HSC-R marcsec       <=                    20.0     13.1
               AM2  HSC-R marcsec       <=                    10.0        0
               AF2  HSC-R               <=                    10.0     4.22
               AD2  HSC-R marcsec       <=                    20.0     14.5
               AM3  HSC-R marcsec       <=                    15.0        0
               AF3  HSC-R               <=                    10.0    0.337
               AD3  HSC-R marcsec       <=                    30.0       11
               PA1  HSC-R    mmag       <=                     5.0     52.1
               PA2  HSC-R    mmag       <=                    15.0      165
               PF1  HSC-R               <=                    10.0     72.3
               TE1  HSC-R               <=                   2e-05 4.78e-06
               TE2  HSC-R               <=                   5e-05 3.38e-07
            ====== ====== ======= ======== ======================= ======== ========
            

            Show
            wmwood-vasey Michael Wood-Vasey added a comment - A final command to add once the JSON output files are generated would be reportPerformance.py . This produces a text file report_performance.rst summarizing the KPMs for each filter. E.g., if the outputs are in JSON files in directories such as w_2018_08_HSC-R/ and w_2018_08_HSC-I , then reportPerformance.py w_2018_08_HSC-*/*.json will create a report_performance.rst file with the following table summary: [wmwv@lsst-dev01 HSC_RC_analysis]$ cat report_performance.rst ====== ====== ======= ======== ======================= ======== ======== Metric Filter Unit Operator SRD Requirement: design Value Comments ====== ====== ======= ======== ======================= ======== ======== AM1 HSC-I marcsec <= 10.0 4.63 AF1 HSC-I <= 10.0 4.4 AD1 HSC-I marcsec <= 20.0 14.4 AM2 HSC-I marcsec <= 10.0 4.12 AF2 HSC-I <= 10.0 4.45 AD2 HSC-I marcsec <= 20.0 14.5 AM3 HSC-I marcsec <= 15.0 0 AF3 HSC-I <= 10.0 0.83 AD3 HSC-I marcsec <= 30.0 12.1 PA1 HSC-I mmag <= 5.0 44 PA2 HSC-I mmag <= 15.0 320 PF1 HSC-I <= 10.0 68.9 TE1 HSC-I <= 2e-05 3.57e-06 TE2 HSC-I <= 5e-05 3.71e-07 AM1 HSC-R marcsec <= 10.0 0 AF1 HSC-R <= 10.0 3.25 AD1 HSC-R marcsec <= 20.0 13.1 AM2 HSC-R marcsec <= 10.0 0 AF2 HSC-R <= 10.0 4.22 AD2 HSC-R marcsec <= 20.0 14.5 AM3 HSC-R marcsec <= 15.0 0 AF3 HSC-R <= 10.0 0.337 AD3 HSC-R marcsec <= 30.0 11 PA1 HSC-R mmag <= 5.0 52.1 PA2 HSC-R mmag <= 15.0 165 PF1 HSC-R <= 10.0 72.3 TE1 HSC-R <= 2e-05 4.78e-06 TE2 HSC-R <= 5e-05 3.38e-07 ====== ====== ======= ======== ======================= ======== ========
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            From running reportPerformance.py with w_2018_10, I got the following

            /software/lsstsw/stack3_20171023/python/miniconda3-4.3.21/lib/python3.6/site-packages/astropy/table/row.py:56: FutureWarning: elementwise == comparison failed and returning scalar instead; this will raise an error or perform elementwise comparison in the future.
              return self.as_void() == other
            /home/hchiang2/stack/validate_drp/python/lsst/validate/drp/report_performance.py:183: UserWarning: Warning: converting a masked element to nan.
              return format_string.format(float(f))
            Metric: TE1 not available in release_metrics file.
            

            Nonetheless the report_performance.rst file was generated:

            ====== ====== ======= ======== ======================= ======== ========
            Metric Filter    Unit Operator SRD Requirement: design    Value Comments
            ====== ====== ======= ======== ======================= ======== ========
               AM1  HSC-G marcsec       <=                    10.0     7.84         
               AF1  HSC-G               <=                    10.0     2.15         
               AD1  HSC-G marcsec       <=                    20.0     8.58         
               AM2  HSC-G marcsec       <=                    10.0     8.22         
               AF2  HSC-G               <=                    10.0     2.03         
               AD2  HSC-G marcsec       <=                    20.0     9.14         
               AM3  HSC-G               <=                    15.0       --         
               AF3  HSC-G               <=                    10.0       --         
               AD3  HSC-G               <=                    30.0       --         
               PA1  HSC-G    mmag       <=                     5.0     17.7         
               PA2  HSC-G    mmag       <=                    15.0     35.1         
               PF1  HSC-G               <=                    10.0     39.4         
               TE1  HSC-G               <=                   2e-05 1.62e-06         
               TE2  HSC-G               <=                   5e-05 1.51e-07         
               AM1  HSC-I marcsec       <=                    10.0      7.9         
               AF1  HSC-I               <=                    10.0     2.19         
               AD1  HSC-I marcsec       <=                    20.0     8.01         
               AM2  HSC-I marcsec       <=                    10.0      8.2         
               AF2  HSC-I               <=                    10.0      3.7         
               AD2  HSC-I marcsec       <=                    20.0     9.84         
               AM3  HSC-I               <=                    15.0       --         
               AF3  HSC-I               <=                    10.0       --         
               AD3  HSC-I               <=                    30.0       --         
               PA1  HSC-I    mmag       <=                     5.0     18.4         
               PA2  HSC-I    mmag       <=                    15.0     38.2         
               PF1  HSC-I               <=                    10.0     42.6         
               TE1  HSC-I               <=                   2e-05 3.77e-06         
               TE2  HSC-I               <=                   5e-05 8.59e-07         
               AM1  HSC-R marcsec       <=                    10.0      9.7         
               AF1  HSC-R               <=                    10.0     4.13         
               AD1  HSC-R marcsec       <=                    20.0     11.1         
               AM2  HSC-R marcsec       <=                    10.0     9.34         
               AF2  HSC-R               <=                    10.0     6.12         
               AD2  HSC-R marcsec       <=                    20.0     13.1         
               AM3  HSC-R               <=                    15.0       --         
               AF3  HSC-R               <=                    10.0       --         
               AD3  HSC-R               <=                    30.0       --         
               PA1  HSC-R    mmag       <=                     5.0     21.7         
               PA2  HSC-R    mmag       <=                    15.0     42.2         
               PF1  HSC-R               <=                    10.0     48.3         
               TE1  HSC-R               <=                   2e-05 5.53e-06         
               TE2  HSC-R               <=                   5e-05 9.41e-08         
               AM1  HSC-Y marcsec       <=                    10.0      3.7         
               AF1  HSC-Y               <=                    10.0     3.71         
               AD1  HSC-Y marcsec       <=                    20.0       12         
               AM2  HSC-Y marcsec       <=                    10.0     6.64         
               AF2  HSC-Y               <=                    10.0     2.79         
               AD2  HSC-Y marcsec       <=                    20.0     11.1         
               AM3  HSC-Y               <=                    15.0       --         
               AF3  HSC-Y               <=                    10.0       --         
               AD3  HSC-Y               <=                    30.0       --         
               PA1  HSC-Y    mmag       <=                     7.5     19.1         
               PA2  HSC-Y    mmag       <=                    22.5       39         
               PF1  HSC-Y               <=                    10.0     28.7         
               TE1  HSC-Y               <=                   2e-05 8.61e-06         
               TE2  HSC-Y               <=                   5e-05 1.93e-07         
               AM1  HSC-Z marcsec       <=                    10.0     6.55         
               AF1  HSC-Z               <=                    10.0     1.61         
               AD1  HSC-Z marcsec       <=                    20.0     8.93         
               AM2  HSC-Z marcsec       <=                    10.0     7.63         
               AF2  HSC-Z               <=                    10.0     1.55         
               AD2  HSC-Z marcsec       <=                    20.0     7.47         
               AM3  HSC-Z               <=                    15.0       --         
               AF3  HSC-Z               <=                    10.0       --         
               AD3  HSC-Z               <=                    30.0       --         
               PA1  HSC-Z    mmag       <=                     7.5     17.9         
               PA2  HSC-Z    mmag       <=                    22.5     34.9         
               PF1  HSC-Z               <=                    10.0       24         
               TE1  HSC-Z               <=                   2e-05 5.55e-06         
               TE2  HSC-Z               <=                   5e-05 1.65e-07         
            ====== ====== ======= ======== ======================= ======== ========
            

            Show
            hchiang2 Hsin-Fang Chiang added a comment - From running  reportPerformance.py with w_2018_10 , I got the following /software/lsstsw/stack3_20171023/python/miniconda3-4.3.21/lib/python3.6/site-packages/astropy/table/row.py:56: FutureWarning: elementwise == comparison failed and returning scalar instead; this will raise an error or perform elementwise comparison in the future. return self.as_void() == other /home/hchiang2/stack/validate_drp/python/lsst/validate/drp/report_performance.py:183: UserWarning: Warning: converting a masked element to nan. return format_string.format(float(f)) Metric: TE1 not available in release_metrics file. Nonetheless the report_performance.rst file was generated: ====== ====== ======= ======== ======================= ======== ======== Metric Filter Unit Operator SRD Requirement: design Value Comments ====== ====== ======= ======== ======================= ======== ======== AM1 HSC-G marcsec <= 10.0 7.84 AF1 HSC-G <= 10.0 2.15 AD1 HSC-G marcsec <= 20.0 8.58 AM2 HSC-G marcsec <= 10.0 8.22 AF2 HSC-G <= 10.0 2.03 AD2 HSC-G marcsec <= 20.0 9.14 AM3 HSC-G <= 15.0 -- AF3 HSC-G <= 10.0 -- AD3 HSC-G <= 30.0 -- PA1 HSC-G mmag <= 5.0 17.7 PA2 HSC-G mmag <= 15.0 35.1 PF1 HSC-G <= 10.0 39.4 TE1 HSC-G <= 2e-05 1.62e-06 TE2 HSC-G <= 5e-05 1.51e-07 AM1 HSC-I marcsec <= 10.0 7.9 AF1 HSC-I <= 10.0 2.19 AD1 HSC-I marcsec <= 20.0 8.01 AM2 HSC-I marcsec <= 10.0 8.2 AF2 HSC-I <= 10.0 3.7 AD2 HSC-I marcsec <= 20.0 9.84 AM3 HSC-I <= 15.0 -- AF3 HSC-I <= 10.0 -- AD3 HSC-I <= 30.0 -- PA1 HSC-I mmag <= 5.0 18.4 PA2 HSC-I mmag <= 15.0 38.2 PF1 HSC-I <= 10.0 42.6 TE1 HSC-I <= 2e-05 3.77e-06 TE2 HSC-I <= 5e-05 8.59e-07 AM1 HSC-R marcsec <= 10.0 9.7 AF1 HSC-R <= 10.0 4.13 AD1 HSC-R marcsec <= 20.0 11.1 AM2 HSC-R marcsec <= 10.0 9.34 AF2 HSC-R <= 10.0 6.12 AD2 HSC-R marcsec <= 20.0 13.1 AM3 HSC-R <= 15.0 -- AF3 HSC-R <= 10.0 -- AD3 HSC-R <= 30.0 -- PA1 HSC-R mmag <= 5.0 21.7 PA2 HSC-R mmag <= 15.0 42.2 PF1 HSC-R <= 10.0 48.3 TE1 HSC-R <= 2e-05 5.53e-06 TE2 HSC-R <= 5e-05 9.41e-08 AM1 HSC-Y marcsec <= 10.0 3.7 AF1 HSC-Y <= 10.0 3.71 AD1 HSC-Y marcsec <= 20.0 12 AM2 HSC-Y marcsec <= 10.0 6.64 AF2 HSC-Y <= 10.0 2.79 AD2 HSC-Y marcsec <= 20.0 11.1 AM3 HSC-Y <= 15.0 -- AF3 HSC-Y <= 10.0 -- AD3 HSC-Y <= 30.0 -- PA1 HSC-Y mmag <= 7.5 19.1 PA2 HSC-Y mmag <= 22.5 39 PF1 HSC-Y <= 10.0 28.7 TE1 HSC-Y <= 2e-05 8.61e-06 TE2 HSC-Y <= 5e-05 1.93e-07 AM1 HSC-Z marcsec <= 10.0 6.55 AF1 HSC-Z <= 10.0 1.61 AD1 HSC-Z marcsec <= 20.0 8.93 AM2 HSC-Z marcsec <= 10.0 7.63 AF2 HSC-Z <= 10.0 1.55 AD2 HSC-Z marcsec <= 20.0 7.47 AM3 HSC-Z <= 15.0 -- AF3 HSC-Z <= 10.0 -- AD3 HSC-Z <= 30.0 -- PA1 HSC-Z mmag <= 7.5 17.9 PA2 HSC-Z mmag <= 22.5 34.9 PF1 HSC-Z <= 10.0 24 TE1 HSC-Z <= 2e-05 5.55e-06 TE2 HSC-Z <= 5e-05 1.65e-07 ====== ====== ======= ======== ======================= ======== ========
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            The results and logs of running the 3 tasks with w_2018_10 are at:
            /project/hsc_rc/w_2018_10/DM-13647/validate_drp_outputs/

            Michael Wood-Vasey please take a look.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - The results and logs of running the 3 tasks with w_2018_10 are at: /project/hsc_rc/w_2018_10/ DM-13647 /validate_drp_outputs/ Michael Wood-Vasey please take a look.
            hchiang2 Hsin-Fang Chiang made changes -
            Resolution Done [ 10000 ]
            Status In Progress [ 3 ] Done [ 10002 ]
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            p.s. The reportPerformance.py above was run with the json files from all 3 tracts. Would we want to separate them per tract?

            Show
            hchiang2 Hsin-Fang Chiang added a comment - p.s. The reportPerformance.py above was run with the json files from all 3 tracts. Would we want to separate them per tract?
            Hide
            wmwood-vasey Michael Wood-Vasey added a comment -

            Sorry, I should have given a heads up that I expected the TE1 not available in release_metrics file. message. It still generates a summary result, so there's some bug somewhere that I should chase down at some point.

            Show
            wmwood-vasey Michael Wood-Vasey added a comment - Sorry, I should have given a heads up that I expected the TE1 not available in release_metrics file. message. It still generates a summary result, so there's some bug somewhere that I should chase down at some point.
            Hide
            wmwood-vasey Michael Wood-Vasey added a comment -

            Yes, separate by tract for now.

            Show
            wmwood-vasey Michael Wood-Vasey added a comment - Yes, separate by tract for now.
            Hide
            wmwood-vasey Michael Wood-Vasey added a comment -

            > The results and logs of running the 3 tasks with w_2018_10 are at:
            > /project/hsc_rc/w_2018_10/DM-13647/validate_drp_outputs/
            >
            > Michael Wood-Vasey please take a look.

            These plots and output logs look good.

            Show
            wmwood-vasey Michael Wood-Vasey added a comment - > The results and logs of running the 3 tasks with w_2018_10 are at: > /project/hsc_rc/w_2018_10/ DM-13647 /validate_drp_outputs/ > > Michael Wood-Vasey please take a look. These plots and output logs look good.
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            The per-tract results are in the same folder now: 

            /project/hsc_rc/w_2018_10/DM-13647/validate_drp_outputs/9615/report_performance-9615.rst
            /project/hsc_rc/w_2018_10/DM-13647/validate_drp_outputs/9697/report_performance-9697.rst
            /project/hsc_rc/w_2018_10/DM-13647/validate_drp_outputs/9813/report_performance-9813.rst

            Show
            hchiang2 Hsin-Fang Chiang added a comment - The per-tract results are in the same folder now:  /project/hsc_rc/w_2018_10/ DM-13647 /validate_drp_outputs/9615/report_performance-9615.rst /project/hsc_rc/w_2018_10/ DM-13647 /validate_drp_outputs/9697/report_performance-9697.rst /project/hsc_rc/w_2018_10/ DM-13647 /validate_drp_outputs/9813/report_performance-9813.rst
            Hide
            wmwood-vasey Michael Wood-Vasey added a comment -

            Show
            wmwood-vasey Michael Wood-Vasey added a comment -
            wmwood-vasey Michael Wood-Vasey made changes -
            Link This issue is triggering DM-13839 [ DM-13839 ]

              People

              Assignee:
              hchiang2 Hsin-Fang Chiang
              Reporter:
              hchiang2 Hsin-Fang Chiang
              Watchers:
              Hsin-Fang Chiang, Michael Wood-Vasey
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.