Details
-
Type:
Story
-
Status: Done
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: None
-
Labels:None
-
Story Points:1
-
Epic Link:
-
Team:Data Facility
Description
Run matchedVisitMetrics.py and validateDrp.py with the HSC RC2 w_2018_10 outputs from DM-13647.
Follow Michael Wood-Vasey's example:
# Need to specify tract, otherwise one will get a meas_base failure.
|
matchedVisitMetrics.py "${REPO}" --output "${OUTREPO}" --config useJointCal=True --id visit=${VISITIDS} tract=${TRACT} |
|
|
# Capture the output
|
validateDrp.py "${OUTREPO}"/matchedVisit_${FILTER}.json --noplot > ${OUTREPO}/metrics.log |
Use the ticket branch of DM-13742.
Attachments
Issue Links
Activity
A final command to add once the JSON output files are generated would be reportPerformance.py. This produces a text file report_performance.rst summarizing the KPMs for each filter. E.g., if the outputs are in JSON files in directories such as w_2018_08_HSC-R/ and w_2018_08_HSC-I, then
reportPerformance.py w_2018_08_HSC-*/*.json
|
will create a report_performance.rst file with the following table summary:
[wmwv@lsst-dev01 HSC_RC_analysis]$ cat report_performance.rst
|
====== ====== ======= ======== ======================= ======== ========
|
Metric Filter Unit Operator SRD Requirement: design Value Comments
|
====== ====== ======= ======== ======================= ======== ========
|
AM1 HSC-I marcsec <= 10.0 4.63
|
AF1 HSC-I <= 10.0 4.4
|
AD1 HSC-I marcsec <= 20.0 14.4
|
AM2 HSC-I marcsec <= 10.0 4.12
|
AF2 HSC-I <= 10.0 4.45
|
AD2 HSC-I marcsec <= 20.0 14.5
|
AM3 HSC-I marcsec <= 15.0 0
|
AF3 HSC-I <= 10.0 0.83
|
AD3 HSC-I marcsec <= 30.0 12.1
|
PA1 HSC-I mmag <= 5.0 44
|
PA2 HSC-I mmag <= 15.0 320
|
PF1 HSC-I <= 10.0 68.9
|
TE1 HSC-I <= 2e-05 3.57e-06
|
TE2 HSC-I <= 5e-05 3.71e-07
|
AM1 HSC-R marcsec <= 10.0 0
|
AF1 HSC-R <= 10.0 3.25
|
AD1 HSC-R marcsec <= 20.0 13.1
|
AM2 HSC-R marcsec <= 10.0 0
|
AF2 HSC-R <= 10.0 4.22
|
AD2 HSC-R marcsec <= 20.0 14.5
|
AM3 HSC-R marcsec <= 15.0 0
|
AF3 HSC-R <= 10.0 0.337
|
AD3 HSC-R marcsec <= 30.0 11
|
PA1 HSC-R mmag <= 5.0 52.1
|
PA2 HSC-R mmag <= 15.0 165
|
PF1 HSC-R <= 10.0 72.3
|
TE1 HSC-R <= 2e-05 4.78e-06
|
TE2 HSC-R <= 5e-05 3.38e-07
|
====== ====== ======= ======== ======================= ======== ========
|
Michael Wood-Vasey
added a comment - A final command to add once the JSON output files are generated would be reportPerformance.py . This produces a text file report_performance.rst summarizing the KPMs for each filter. E.g., if the outputs are in JSON files in directories such as w_2018_08_HSC-R/ and w_2018_08_HSC-I , then
reportPerformance.py w_2018_08_HSC-*/*.json
will create a report_performance.rst file with the following table summary:
[wmwv@lsst-dev01 HSC_RC_analysis]$ cat report_performance.rst
====== ====== ======= ======== ======================= ======== ========
Metric Filter Unit Operator SRD Requirement: design Value Comments
====== ====== ======= ======== ======================= ======== ========
AM1 HSC-I marcsec <= 10.0 4.63
AF1 HSC-I <= 10.0 4.4
AD1 HSC-I marcsec <= 20.0 14.4
AM2 HSC-I marcsec <= 10.0 4.12
AF2 HSC-I <= 10.0 4.45
AD2 HSC-I marcsec <= 20.0 14.5
AM3 HSC-I marcsec <= 15.0 0
AF3 HSC-I <= 10.0 0.83
AD3 HSC-I marcsec <= 30.0 12.1
PA1 HSC-I mmag <= 5.0 44
PA2 HSC-I mmag <= 15.0 320
PF1 HSC-I <= 10.0 68.9
TE1 HSC-I <= 2e-05 3.57e-06
TE2 HSC-I <= 5e-05 3.71e-07
AM1 HSC-R marcsec <= 10.0 0
AF1 HSC-R <= 10.0 3.25
AD1 HSC-R marcsec <= 20.0 13.1
AM2 HSC-R marcsec <= 10.0 0
AF2 HSC-R <= 10.0 4.22
AD2 HSC-R marcsec <= 20.0 14.5
AM3 HSC-R marcsec <= 15.0 0
AF3 HSC-R <= 10.0 0.337
AD3 HSC-R marcsec <= 30.0 11
PA1 HSC-R mmag <= 5.0 52.1
PA2 HSC-R mmag <= 15.0 165
PF1 HSC-R <= 10.0 72.3
TE1 HSC-R <= 2e-05 4.78e-06
TE2 HSC-R <= 5e-05 3.38e-07
====== ====== ======= ======== ======================= ======== ========
From running reportPerformance.py with w_2018_10, I got the following
/software/lsstsw/stack3_20171023/python/miniconda3-4.3.21/lib/python3.6/site-packages/astropy/table/row.py:56: FutureWarning: elementwise == comparison failed and returning scalar instead; this will raise an error or perform elementwise comparison in the future.
|
return self.as_void() == other
|
/home/hchiang2/stack/validate_drp/python/lsst/validate/drp/report_performance.py:183: UserWarning: Warning: converting a masked element to nan.
|
return format_string.format(float(f))
|
Metric: TE1 not available in release_metrics file.
|
Nonetheless the report_performance.rst file was generated:
====== ====== ======= ======== ======================= ======== ========
|
Metric Filter Unit Operator SRD Requirement: design Value Comments
|
====== ====== ======= ======== ======================= ======== ========
|
AM1 HSC-G marcsec <= 10.0 7.84
|
AF1 HSC-G <= 10.0 2.15
|
AD1 HSC-G marcsec <= 20.0 8.58
|
AM2 HSC-G marcsec <= 10.0 8.22
|
AF2 HSC-G <= 10.0 2.03
|
AD2 HSC-G marcsec <= 20.0 9.14
|
AM3 HSC-G <= 15.0 --
|
AF3 HSC-G <= 10.0 --
|
AD3 HSC-G <= 30.0 --
|
PA1 HSC-G mmag <= 5.0 17.7
|
PA2 HSC-G mmag <= 15.0 35.1
|
PF1 HSC-G <= 10.0 39.4
|
TE1 HSC-G <= 2e-05 1.62e-06
|
TE2 HSC-G <= 5e-05 1.51e-07
|
AM1 HSC-I marcsec <= 10.0 7.9
|
AF1 HSC-I <= 10.0 2.19
|
AD1 HSC-I marcsec <= 20.0 8.01
|
AM2 HSC-I marcsec <= 10.0 8.2
|
AF2 HSC-I <= 10.0 3.7
|
AD2 HSC-I marcsec <= 20.0 9.84
|
AM3 HSC-I <= 15.0 --
|
AF3 HSC-I <= 10.0 --
|
AD3 HSC-I <= 30.0 --
|
PA1 HSC-I mmag <= 5.0 18.4
|
PA2 HSC-I mmag <= 15.0 38.2
|
PF1 HSC-I <= 10.0 42.6
|
TE1 HSC-I <= 2e-05 3.77e-06
|
TE2 HSC-I <= 5e-05 8.59e-07
|
AM1 HSC-R marcsec <= 10.0 9.7
|
AF1 HSC-R <= 10.0 4.13
|
AD1 HSC-R marcsec <= 20.0 11.1
|
AM2 HSC-R marcsec <= 10.0 9.34
|
AF2 HSC-R <= 10.0 6.12
|
AD2 HSC-R marcsec <= 20.0 13.1
|
AM3 HSC-R <= 15.0 --
|
AF3 HSC-R <= 10.0 --
|
AD3 HSC-R <= 30.0 --
|
PA1 HSC-R mmag <= 5.0 21.7
|
PA2 HSC-R mmag <= 15.0 42.2
|
PF1 HSC-R <= 10.0 48.3
|
TE1 HSC-R <= 2e-05 5.53e-06
|
TE2 HSC-R <= 5e-05 9.41e-08
|
AM1 HSC-Y marcsec <= 10.0 3.7
|
AF1 HSC-Y <= 10.0 3.71
|
AD1 HSC-Y marcsec <= 20.0 12
|
AM2 HSC-Y marcsec <= 10.0 6.64
|
AF2 HSC-Y <= 10.0 2.79
|
AD2 HSC-Y marcsec <= 20.0 11.1
|
AM3 HSC-Y <= 15.0 --
|
AF3 HSC-Y <= 10.0 --
|
AD3 HSC-Y <= 30.0 --
|
PA1 HSC-Y mmag <= 7.5 19.1
|
PA2 HSC-Y mmag <= 22.5 39
|
PF1 HSC-Y <= 10.0 28.7
|
TE1 HSC-Y <= 2e-05 8.61e-06
|
TE2 HSC-Y <= 5e-05 1.93e-07
|
AM1 HSC-Z marcsec <= 10.0 6.55
|
AF1 HSC-Z <= 10.0 1.61
|
AD1 HSC-Z marcsec <= 20.0 8.93
|
AM2 HSC-Z marcsec <= 10.0 7.63
|
AF2 HSC-Z <= 10.0 1.55
|
AD2 HSC-Z marcsec <= 20.0 7.47
|
AM3 HSC-Z <= 15.0 --
|
AF3 HSC-Z <= 10.0 --
|
AD3 HSC-Z <= 30.0 --
|
PA1 HSC-Z mmag <= 7.5 17.9
|
PA2 HSC-Z mmag <= 22.5 34.9
|
PF1 HSC-Z <= 10.0 24
|
TE1 HSC-Z <= 2e-05 5.55e-06
|
TE2 HSC-Z <= 5e-05 1.65e-07
|
====== ====== ======= ======== ======================= ======== ========
|
Hsin-Fang Chiang
added a comment - From running reportPerformance.py with w_2018_10 , I got the following
/software/lsstsw/stack3_20171023/python/miniconda3-4.3.21/lib/python3.6/site-packages/astropy/table/row.py:56: FutureWarning: elementwise == comparison failed and returning scalar instead; this will raise an error or perform elementwise comparison in the future.
return self.as_void() == other
/home/hchiang2/stack/validate_drp/python/lsst/validate/drp/report_performance.py:183: UserWarning: Warning: converting a masked element to nan.
return format_string.format(float(f))
Metric: TE1 not available in release_metrics file.
Nonetheless the report_performance.rst file was generated:
====== ====== ======= ======== ======================= ======== ========
Metric Filter Unit Operator SRD Requirement: design Value Comments
====== ====== ======= ======== ======================= ======== ========
AM1 HSC-G marcsec <= 10.0 7.84
AF1 HSC-G <= 10.0 2.15
AD1 HSC-G marcsec <= 20.0 8.58
AM2 HSC-G marcsec <= 10.0 8.22
AF2 HSC-G <= 10.0 2.03
AD2 HSC-G marcsec <= 20.0 9.14
AM3 HSC-G <= 15.0 --
AF3 HSC-G <= 10.0 --
AD3 HSC-G <= 30.0 --
PA1 HSC-G mmag <= 5.0 17.7
PA2 HSC-G mmag <= 15.0 35.1
PF1 HSC-G <= 10.0 39.4
TE1 HSC-G <= 2e-05 1.62e-06
TE2 HSC-G <= 5e-05 1.51e-07
AM1 HSC-I marcsec <= 10.0 7.9
AF1 HSC-I <= 10.0 2.19
AD1 HSC-I marcsec <= 20.0 8.01
AM2 HSC-I marcsec <= 10.0 8.2
AF2 HSC-I <= 10.0 3.7
AD2 HSC-I marcsec <= 20.0 9.84
AM3 HSC-I <= 15.0 --
AF3 HSC-I <= 10.0 --
AD3 HSC-I <= 30.0 --
PA1 HSC-I mmag <= 5.0 18.4
PA2 HSC-I mmag <= 15.0 38.2
PF1 HSC-I <= 10.0 42.6
TE1 HSC-I <= 2e-05 3.77e-06
TE2 HSC-I <= 5e-05 8.59e-07
AM1 HSC-R marcsec <= 10.0 9.7
AF1 HSC-R <= 10.0 4.13
AD1 HSC-R marcsec <= 20.0 11.1
AM2 HSC-R marcsec <= 10.0 9.34
AF2 HSC-R <= 10.0 6.12
AD2 HSC-R marcsec <= 20.0 13.1
AM3 HSC-R <= 15.0 --
AF3 HSC-R <= 10.0 --
AD3 HSC-R <= 30.0 --
PA1 HSC-R mmag <= 5.0 21.7
PA2 HSC-R mmag <= 15.0 42.2
PF1 HSC-R <= 10.0 48.3
TE1 HSC-R <= 2e-05 5.53e-06
TE2 HSC-R <= 5e-05 9.41e-08
AM1 HSC-Y marcsec <= 10.0 3.7
AF1 HSC-Y <= 10.0 3.71
AD1 HSC-Y marcsec <= 20.0 12
AM2 HSC-Y marcsec <= 10.0 6.64
AF2 HSC-Y <= 10.0 2.79
AD2 HSC-Y marcsec <= 20.0 11.1
AM3 HSC-Y <= 15.0 --
AF3 HSC-Y <= 10.0 --
AD3 HSC-Y <= 30.0 --
PA1 HSC-Y mmag <= 7.5 19.1
PA2 HSC-Y mmag <= 22.5 39
PF1 HSC-Y <= 10.0 28.7
TE1 HSC-Y <= 2e-05 8.61e-06
TE2 HSC-Y <= 5e-05 1.93e-07
AM1 HSC-Z marcsec <= 10.0 6.55
AF1 HSC-Z <= 10.0 1.61
AD1 HSC-Z marcsec <= 20.0 8.93
AM2 HSC-Z marcsec <= 10.0 7.63
AF2 HSC-Z <= 10.0 1.55
AD2 HSC-Z marcsec <= 20.0 7.47
AM3 HSC-Z <= 15.0 --
AF3 HSC-Z <= 10.0 --
AD3 HSC-Z <= 30.0 --
PA1 HSC-Z mmag <= 7.5 17.9
PA2 HSC-Z mmag <= 22.5 34.9
PF1 HSC-Z <= 10.0 24
TE1 HSC-Z <= 2e-05 5.55e-06
TE2 HSC-Z <= 5e-05 1.65e-07
====== ====== ======= ======== ======================= ======== ========
The results and logs of running the 3 tasks with w_2018_10 are at:
/project/hsc_rc/w_2018_10/DM-13647/validate_drp_outputs/
Michael Wood-Vasey please take a look.
Hsin-Fang Chiang
added a comment - The results and logs of running the 3 tasks with w_2018_10 are at:
/project/hsc_rc/w_2018_10/ DM-13647 /validate_drp_outputs/
Michael Wood-Vasey please take a look. p.s. The reportPerformance.py above was run with the json files from all 3 tracts. Would we want to separate them per tract?
Hsin-Fang Chiang
added a comment - p.s. The reportPerformance.py above was run with the json files from all 3 tracts. Would we want to separate them per tract? Sorry, I should have given a heads up that I expected the TE1 not available in release_metrics file. message. It still generates a summary result, so there's some bug somewhere that I should chase down at some point.
Michael Wood-Vasey
added a comment - Sorry, I should have given a heads up that I expected the TE1 not available in release_metrics file. message. It still generates a summary result, so there's some bug somewhere that I should chase down at some point. Yes, separate by tract for now.
Michael Wood-Vasey
added a comment - Yes, separate by tract for now. > The results and logs of running the 3 tasks with w_2018_10 are at:
> /project/hsc_rc/w_2018_10/DM-13647/validate_drp_outputs/
>
> Michael Wood-Vasey please take a look.
These plots and output logs look good.
Michael Wood-Vasey
added a comment -
> The results and logs of running the 3 tasks with w_2018_10 are at:
> /project/hsc_rc/w_2018_10/ DM-13647 /validate_drp_outputs/
>
> Michael Wood-Vasey please take a look.
These plots and output logs look good. The per-tract results are in the same folder now:
/project/hsc_rc/w_2018_10/DM-13647/validate_drp_outputs/9615/report_performance-9615.rst
/project/hsc_rc/w_2018_10/DM-13647/validate_drp_outputs/9697/report_performance-9697.rst
/project/hsc_rc/w_2018_10/DM-13647/validate_drp_outputs/9813/report_performance-9813.rst
Hsin-Fang Chiang
added a comment - The per-tract results are in the same folder now:
/project/hsc_rc/w_2018_10/ DM-13647 /validate_drp_outputs/9615/report_performance-9615.rst
/project/hsc_rc/w_2018_10/ DM-13647 /validate_drp_outputs/9697/report_performance-9697.rst
/project/hsc_rc/w_2018_10/ DM-13647 /validate_drp_outputs/9813/report_performance-9813.rst 
Michael Wood-Vasey
added a comment - 
The HSC RC2 dataset includes data in filter=NB0921 in the COSMOS tract; it will be ignored here because filter metric definitions need to be added.
File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_base/14.0-6-ge2c9487+51/python/lsst/pipe/base/cmdLineTask.py", line 586, in parseAndRunresultList = taskRunner.run(parsedCmd)File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/pipe_base/14.0-6-ge2c9487+51/python/lsst/pipe/base/cmdLineTask.py", line 240, in runresultList = list(mapFunc(self, targetList))File "/home/hchiang2/stack/validate_drp/python/lsst/validate/drp/matchedVisitMetricsTask.py", line 36, in __call__return task.run(*args)File "/home/hchiang2/stack/validate_drp/python/lsst/validate/drp/matchedVisitMetricsTask.py", line 133, in runverbose=self.config.verbose)File "/home/hchiang2/stack/validate_drp/python/lsst/validate/drp/validate.py", line 311, in runOneFilterjob=job, linkedBlobs=linkedBlobs)File "/home/hchiang2/stack/validate_drp/python/lsst/validate/drp/calcsrd/pf1.py", line 72, in __init__PA2.get_spec(spec_name, filter_name=self.filter_name)File "/software/lsstsw/stack3_20171023/stack/miniconda3-4.3.21-10a4fa6/Linux64/validate_base/master-g24db930f3a+22/python/lsst/validate/base/metric.py", line 381, in get_specname, filter_name, self.name))RuntimeError: No PA2 spec found for name=stretch filter_name=NB0921