Details
-
Type:
Story
-
Status: Won't Fix
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: Validation
-
Labels:
-
Epic Link:
-
Team:SQuaRE
Description
After the validate_base shift, validateDrp.py now requires a filter to be listed in the etc/metrics.yaml in order for it to run. If no match is found, it bails out.
1. validateDrp.py should instead do "something reasonable". Likely this means picking 'r' with a clear message that it is doing so.
2. validateDrp.py should successfully calculate all metrics that it can. Failing on one metric or with the ocnfiguration of same should not lead to crashing:
[wmwv@lsst-dev01 tmp]$ validateDrp.py DATA/rerun/20161216
|
CameraMapper WARN: Calibration root directory not found: DATA/rerun/20161216/CALIB
|
CameraMapper WARN: Calibration root directory not found: DATA/rerun/20161216/CALIB
|
1077 sources in ccd 16 visit 903986
|
953 sources in ccd 24 visit 903988
|
1174 sources in ccd 18 visit 903990
|
1049 sources in ccd 23 visit 903986
|
1140 sources in ccd 25 visit 903990
|
932 sources in ccd 23 visit 903988
|
916 sources in ccd 17 visit 903988
|
778 sources in ccd 22 visit 903986
|
683 sources in ccd 100 visit 903986
|
938 sources in ccd 16 visit 903988
|
Photometric scatter (median) - SNR > 100.0 : 6.9 mmag
|
/nfs/home/wmwv/local/lsst/validate_drp/python/lsst/validate/drp/photerrmodel.py:74: RuntimeWarning: invalid value encountered in sqrt
|
return np.sqrt(sigmaSq)
|
Median value of the astrometric scatter - all magnitudes: 48.763 marcsec
|
Astrometric scatter (median) - snr > 100.0 : 9.6 marcsec
|
No stars found that are 199.0 arcmin--201.0 arcmin apart.
|
Traceback (most recent call last):
|
File "/nfs/home/wmwv/local/lsst/validate_drp/bin/validateDrp.py", line 95, in <module>
|
validate.run(args.repo, **kwargs)
|
File "/nfs/home/wmwv/local/lsst/validate_drp/python/lsst/validate/drp/validate.py", line 104, in run
|
**kwargs)
|
File "/nfs/home/wmwv/local/lsst/validate_drp/python/lsst/validate/drp/validate.py", line 242, in runOneFilter
|
job=job, linkedBlobs=linkedBlobs)
|
File "/nfs/home/wmwv/local/lsst/validate_drp/python/lsst/validate/drp/calcsrd/pf1.py", line 72, in __init__
|
PA2.get_spec(spec_name, filter_name=self.filter_name)
|
File "/nfs/home/wmwv/local/lsst/validate_base/python/lsst/validate/base/metric.py", line 346, in get_spec
|
name, filter_name, self.name))
|
RuntimeError: No PA2 spec found for name=stretch filter_name=HSC-I
|
The spirit of validate_drp is that it should just run on a data repo and produce reasonable input. An additional level is providing specific comparison against LSST SRD-level performance requirements or specific other performance levels.
Attachments
Issue Links
Activity
So we're forced to have filters for specifications because validate_drp needs to be able check metrics as specified in the SRD, which of course have filter dependence. An additional complication with PA2 is that it needs the (filter-dependent) specification level of PF1 in order to calculate it (no way around this; it's how PA2 in the SRD is written).
What we can do is have a way of marking a specification level in a metrics.yaml as the default. (Perhaps by having a specification with no filter dependence; this would be used as a fall-back if the filter names do not match).
In the meantime we should probably add HSC-* explicitly to the filter lists for certain specifications in metrics.yaml, given the importance of HSC as a DM-supported camera. That's much better than trying to be implicit and automagically mapping HSC-R to, say, r, behind the scenes.
And you're of course right that the exception should be handled so that validate_drp keeps going even if there's a failure. We can probably add an exception handler wrapper around each measurement attempt.
Yeah, I'm adding the HSC-* filters as part of DM-6328.
Michael Wood-Vasey
added a comment - Yeah, I'm adding the HSC-* filters as part of DM-6328 . Calculating pass/fail statistics of course requires specific thresholds, which are filter dependent.
But calculating the simple metrics such as PA1, AM1 can be done, even if there isn't a level to compare them to.
Dependent metrics such as PA2 would benefit from a reasonable default. I like the idea of a specification with no filter dependence as the default, with per-filter specifications providing overrides. I will leave it to you, Jonathan Sick, to design the format for this.
Michael Wood-Vasey
added a comment - Calculating pass/fail statistics of course requires specific thresholds, which are filter dependent.
But calculating the simple metrics such as PA1, AM1 can be done, even if there isn't a level to compare them to.
Dependent metrics such as PA2 would benefit from a reasonable default. I like the idea of a specification with no filter dependence as the default, with per-filter specifications providing overrides. I will leave it to you, Jonathan Sick , to design the format for this. Closing this as “Won't Fix”, on the basis that validate_drp is being replaced.
This was specifically triggered by looking at HSC data, where filters are named things like 'HSC-R' and 'HSC-I'. While specific rules could be written for those filters, the general point remains. Reasonable defaults should exist and be used.