Details
-
Type:
Bug
-
Status: Won't Fix
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: Validation
-
Labels:None
-
Team:SQuaRE
Description
I ran validate_drp's examples/runCfhtQuickTest.sh several times to compare a branch to master where I expected no change. I was surprised to find a change in PA1, PA2 and PF1 (though everything else was identical) so I ran a twice more, once on the branch and once on master. Each run gave different results. For example:
PA1 : 10.10 mmag < 5.00 mmag == False
|
PF1 : 12.00 % < 10.00 % == False
|
vs.
|
PA1 : 10.35 mmag < 5.00 mmag == False
|
PF1 : 4.00 % < 10.00 % == True
|
for the two runs using master.
Attachments
Issue Links
- relates to
-
DM-7309 Port obs_cfht to Python 3
- Done
Activity
Yes. There is some randomness in the generation of the statistics. This is largely correct behavior.
In brief, the issue is that the repeatability KPMs are evaluated using a selection of random pairs to do the comparisons.
It really is only an issue at low N, characteristic of the small test sample datasets.
Michael Wood-Vasey
added a comment - Yes. There is some randomness in the generation of the statistics. This is largely correct behavior.
In brief, the issue is that the repeatability KPMs are evaluated using a selection of random pairs to do the comparisons.
It really is only an issue at low N, characteristic of the small test sample datasets.
Randomness expected at low N due to choice of random pairs used to compare.
Michael Wood-Vasey
added a comment - Randomness expected at low N due to choice of random pairs used to compare. 
Michael Wood-Vasey requested some files from a full run. I have attached two zip archives containing the requested outputs from two different run of examples/runCfhtTest.sh >runCfhtTest.log using master. Each archive contains: