# Some validate_drp measurements are not reproducible

XMLWordPrintable

#### Details

• Type: Bug
• Status: Won't Fix
• Resolution: Done
• Fix Version/s: None
• Component/s:
• Labels:
None
• Team:
SQuaRE

#### Description

I ran validate_drp's examples/runCfhtQuickTest.sh several times to compare a branch to master where I expected no change. I was surprised to find a change in PA1, PA2 and PF1 (though everything else was identical) so I ran a twice more, once on the branch and once on master. Each run gave different results. For example:

 PA1 : 10.10 mmag < 5.00 mmag == False PF1 : 12.00 % < 10.00 % == False vs. PA1 : 10.35 mmag < 5.00 mmag == False PF1 : 4.00 % < 10.00 % == True 

for the two runs using master.

#### Attachments

1. PAandlog1.zip
93 kB
2. PAandlog2.zip
92 kB

#### Activity

Hide
Russell Owen added a comment -

Michael Wood-Vasey [X] requested some files from a full run. I have attached two zip archives containing the requested outputs from two different run of examples/runCfhtTest.sh >runCfhtTest.log using master. Each archive contains:

• runCfhtTest.log
• Cfht_output_r_PA1.json
• Cfht_output_r_PA1.png
• Cfht_output_r_PA2.json
Show
Russell Owen added a comment - Michael Wood-Vasey [X] requested some files from a full run. I have attached two zip archives containing the requested outputs from two different run of examples/runCfhtTest.sh >runCfhtTest.log using master. Each archive contains: runCfhtTest.log Cfht_output_r_PA1.json Cfht_output_r_PA1.png Cfht_output_r_PA2.json
Hide
Michael Wood-Vasey [X] (Inactive) added a comment -

Yes. There is some randomness in the generation of the statistics. This is largely correct behavior.

In brief, the issue is that the repeatability KPMs are evaluated using a selection of random pairs to do the comparisons.

It really is only an issue at low N, characteristic of the small test sample datasets.

Show
Michael Wood-Vasey [X] (Inactive) added a comment - Yes. There is some randomness in the generation of the statistics. This is largely correct behavior. In brief, the issue is that the repeatability KPMs are evaluated using a selection of random pairs to do the comparisons. It really is only an issue at low N, characteristic of the small test sample datasets.
Hide
Michael Wood-Vasey [X] (Inactive) added a comment -

Randomness expected at low N due to choice of random pairs used to compare.

Show
Michael Wood-Vasey [X] (Inactive) added a comment - Randomness expected at low N due to choice of random pairs used to compare.

#### People

Assignee:
Michael Wood-Vasey [X] (Inactive)
Reporter:
Russell Owen
Watchers:
Michael Wood-Vasey [X] (Inactive), Russell Owen