Fix Version/s: None
Once the validation datasets have been processed, analyze the outputs for completeness. This will include categorizing the failure modes (if any exist ).
Things to be collected could include:
2. Num ref/num source
3. Rms scatter and error on that value
4. Broad categories of failures
- is blocked by
DM-10396 Run dataset for matcher validation
Created ascii files for each of the processed data from HSC, CFHTLS, and HITS. These data are stored in the directory:
2 files are stored per rerun: The visits/ccds where both matcher "succeeded" (as in produced a calExp) and where some part of the pipeline failed (i.e. the visti/ccd contains bad data or at least one of the matchers failed).
These data were created from the output files produced by singleFrameDriver.py by using the Jupyter notebook in "/home/morriscb/Parse and store output logs for matcher.ipynb".
This script could be more generailized for end to end creation of these logs files given a request from the reviewer.
Can you remind me what you mean by fail? I thought it was something more subtle than just failing to produce a calexp. I.e. the match scatter being too large would indicate failure, but maybe that comes in the analysis step.
I think we need to categorize the cases where a failure to produce a calexp happened. What were the causes of those failures? I think that needs to be part of this ticket.
By fail in this context I mean did not produce a calexp for one of 2 cases: 1) Either matcher failed to find any match 2) an earlier pipeline step failed for some reason.
The number of pipeline failures for each of the different reruns are:
CFHTLS_u: 2016 visits, 7 pipeline failures
CFHTLS_i: 2016 visits, 7 pipeline failures
CFHTLS_z: 2016 visits, 7 pipeline failures
CFHTLS: 11,700 visits, 2757 pipeline failures
HITS: 11,040 visits, 162 pipeline failures
HSC: 3952 visits, 0 pipeline failures.
A failure to match either catastrophically or with a bad scatter will be investigated in the next ticket
Started collecting summary statics from data run with both optimistic and pessimistic matchers..