# Compare the jointcal SkyWcs's of the gen2 vs. gen3 w_2022_04 RC2 runs

XMLWordPrintable

#### Details

• Type: Story
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s: None
• Labels:
• Story Points:
2
• Sprint:
DRP S22A
• Team:
Data Release Production
• Urgent?:
No

#### Description

As promised on DM-29821, this ticket is to compare the jointcal external SkyWcs objects from the HSC-RC2 w_2022_04 processing runs: Gen2 (DM-33457) vs. Gen3 (DM-33402). As noted here, the Gen2 run used a special-purpose config override to best match the current Gen3 configs. With the fixes on DM-29008 we are expecting very tight comparisons (with a likely small residual difference stemming from a difference in the order of inputs between Gen2 & Gen3).

#### Activity

Hide
Lauren MacArthur added a comment - - edited

The speculations in the description indeed prove true.  The differences are down to levels so low (typically of order mean difference[stdev] ~0.001[0.02] mas for S/N>100 stars, and this is down by large factors of ~10-100 from the differences in the previous w_2021_46 runs) that I don't think they are of any scientific interest.  These remaining differences are almost certainly due to differences in the order of inputs and are not worth further investigation (noting that it will be up to the Gen3 version to provide the ability to order its inputs deterministically).  As such, I am ready to pull the plug on Gen2 support for jointcal.

For a few more details, the following is our worst case in terms of the mean Gen2/Gen3 difference in Delta(RA)*cos(Dec) for S/N>100 stars (and note I did also look at the individual Delta(RA) and Delta(Dec) but did not notice any distinctions worth any scrutiny):

This mean[stdev] difference is down from 2.80[1.15] mas in the previous w_2021_46 runs.

We also see some that look to have some stochasticity in them (but do note the 1e-7 on the y-axis scale!):

(note the stochasticity is not detector-specific, but rather they are spread throughout the focal plane).

And a handful of intra-detector gradients:

Overall, even our seemingly worst-cases are at levels so low as to be uninteresting.

Show
Lauren MacArthur added a comment - - edited The speculations in the description indeed prove true.  The differences are down to levels so low (typically of order mean difference [stdev] ~0.001 [0.02]  mas for S/N>100 stars, and this is down by large factors of ~10-100 from the differences in the previous w_2021_46  runs) that I don't think they are of any scientific interest.  These remaining differences are almost certainly due to differences in the order of inputs and are not worth further investigation (noting that it will be up to the Gen3 version to provide the ability to order its inputs deterministically).  As such, I am ready to pull the plug on Gen2 support for jointcal . For a few more details, the following is our worst case in terms of the mean Gen2/Gen3 difference in Delta(RA)*cos(Dec) for S/N>100 stars (and note I did also look at the individual Delta(RA) and Delta(Dec) but did not notice any distinctions worth any scrutiny): This mean [stdev] difference is down from 2.80 [1.15] mas in the previous w_2021_46 runs. We also see some that look to have some stochasticity in them (but do note the 1e-7 on the y-axis scale!): (note the stochasticity is not detector-specific, but rather they are spread throughout the focal plane). And a handful of intra-detector gradients: Overall, even our seemingly worst-cases are at levels so low as to be uninteresting.
Hide
Lauren MacArthur added a comment - - edited

As far as how the above visit-level differences propagate into coadd differences, the effects are again very small (differences typically of order mean[stdev] ~0.003[0.06] mas for S/N~>500 stars, and again down by large factors of ~10 from the differences in the previous w_2021_46 runs)

Here is our worst case scenario:

and a more typical case:

See DM-29820 if interested in the differences in the Stellar Locus plots.

Again, this looks overall, to be small enough as to be uninteresting.

Show
Lauren MacArthur added a comment - - edited As far as how the above visit-level differences propagate into coadd differences, the effects are again very small (differences typically of order mean [stdev] ~0.003 [0.06]  mas for S/N~>500 stars, and again down by large factors of ~10 from the differences in the previous w_2021_46  runs) Here is our worst case scenario: and a more typical case: See DM-29820 if interested in the differences in the Stellar Locus plots. Again, this looks overall, to be small enough as to be uninteresting.
Hide
Lauren MacArthur added a comment -

With the above results, I am prepared to pull the plug on Gen2 support for jointcal, but
let me know what you think. If there are any particular cases you want looked at, let me know, or feel free to poke around yourself at:
Gen2:
https://lsst.ncsa.illinois.edu/~lauren/HSC_RC2/w_2022_04/plots/
Gen3:
https://lsst.ncsa.illinois.edu/~lauren/HSC_RC2/w_2022_04/vsGen3/plots/
(it is the latter that has the direct comparison plots).

Show
Lauren MacArthur added a comment - With the above results, I am prepared to pull the plug on Gen2 support for jointcal , but let me know what you think. If there are any particular cases you want looked at, let me know, or feel free to poke around yourself at: Gen2: https://lsst.ncsa.illinois.edu/~lauren/HSC_RC2/w_2022_04/plots/ Gen3: https://lsst.ncsa.illinois.edu/~lauren/HSC_RC2/w_2022_04/vsGen3/plots/ (it is the latter that has the direct comparison plots).
Hide
John Parejko added a comment -

Jointcal input ordering is something I wondered about from early on (see DM-9393). It's a small effect, and I could believe the difference you see here is entirely due to that. When the stars are associated across visits, the exact details of the cross matches can be slightly different, but that difference should come out by the end of the fit. Can you give me links to the logs for a couple of matched gen2/gen3 runs from this comparison? I'm curious whether the final number of stars in the fit and the final chi2 and ndof values are the same or slightly different? If the fitting process is perfect, I'd expect them to be identical (any ambiguous matches would have been removed), but HSC is now using astrometryOutlierRelativeTolerance, which means the fit ends before it's fully converged (to reduce the runtime). I'd bet that that accounts for most (all?) of the difference you see here. I don't think it's worth exploring what gen2/gen3 differences on HSC data look like with that config set to None, since it's so small, but a quick check of the final gen2/gen3 chi2 and ndof, values might be informative, as to how different jointcal thinks it is in those two cases.

This also reminds me that jointcal doesn't include a final count of the number of fittedStars/measuredStars. Those get incorporated into ndof, so it's probably not relevant most of the time.

Show
John Parejko added a comment - Jointcal input ordering is something I wondered about from early on (see DM-9393 ). It's a small effect, and I could believe the difference you see here is entirely due to that. When the stars are associated across visits, the exact details of the cross matches can be slightly different, but that difference should come out by the end of the fit. Can you give me links to the logs for a couple of matched gen2/gen3 runs from this comparison? I'm curious whether the final number of stars in the fit and the final chi2 and ndof values are the same or slightly different? If the fitting process is perfect, I'd expect them to be identical (any ambiguous matches would have been removed), but HSC is now using astrometryOutlierRelativeTolerance , which means the fit ends before it's fully converged (to reduce the runtime). I'd bet that that accounts for most (all?) of the difference you see here. I don't think it's worth exploring what gen2/gen3 differences on HSC data look like with that config set to None, since it's so small, but a quick check of the final gen2/gen3 chi2 and ndof, values might be informative, as to how different jointcal thinks it is in those two cases. This also reminds me that jointcal doesn't include a final count of the number of fittedStars/measuredStars. Those get incorporated into ndof, so it's probably not relevant most of the time.
Hide
Lauren MacArthur added a comment - - edited

Great, thanks John!

The logs are located on lsst-devl at:

  Gen2: /datasets/hsc/repo/rerun/RC/w_2022_04/DM-33457/logs/jointcal/  Gen3: /scratch/brendal4/bps-gen3-rc2/w_2022_04/submit/HSC/runs/RC2/w_2022_04/DM-33402/20220128T212035Z/jobs/jointcal/ 

The logs are not terribly easy to parse & compare, so as an example, I pulled out all the lines with chi2 &/or ndof in them and compare the Gen2 vs. Gen3 versions directly. So, for

 /datasets/hsc/repo/rerun/RC/w_2022_04/DM-33457/logs/jointcal/jointcal9813-HSC-I-59795.log 

vs.

 /scratch/brendal4/bps-gen3-rc2/w_2022_04/submit/HSC/runs/RC2/w_2022_04/DM-33402/20220128T212035Z/jobs/jointcal/9813/HSC-I/e6e6d88e-f483-4041-98ef-8390437230c7_jointcal_9813_.9754984.err 

the corresponding lines are:

 Gen2: Initial chi2/ndof : 4.21833e+16/490498=8.60009e+10 Gen3: Initial chi2/ndof : 4.21833e+16/490498=8.60009e+10   Gen2: Initialize DistortionsVisit chi2/ndof : 1.87784e+08/488122=384.707 Gen3: Initialize DistortionsVisit chi2/ndof : 1.87614e+08/488122=384.358   Gen2: Initialize Distortions chi2/ndof : 9.44308e+06/487510=19.37 Gen3: Initialize Distortions chi2/ndof : 9.27373e+06/487510=19.0227   Gen2: Initialize Positions chi2/ndof : 1.96279e+06/454112=4.32226 Gen3: Initialize Positions chi2/ndof : 1.86372e+06/454112=4.10409   Gen2: Initialize DistortionsPositions chi2/ndof : 1.62282e+06/451124=3.59729 Gen3: Initialize DistortionsPositions chi2/ndof : 1.52377e+06/451124=3.37773   Gen2: Iterations stopped with chi2 cut at 81.596 and relative change of -0.0947015 Gen3: Iterations stopped with chi2 cut at 81.7825 and relative change of -0.109253   Gen2: Fit iteration 0 chi2/ndof : 803915/450176=1.78578 Gen3: Fit iteration 0 chi2/ndof : 799876/450090=1.77715   Gen2: Iterations stopped with chi2 cut at 23.7586 and relative change of -0.00395254 Gen3: Iterations stopped with chi2 cut at 23.9852 and relative change of -0.00407391   Gen2: Fit completed chi2/ndof : 611196/447240=1.36659 Gen3: Fit completed chi2/ndof : 612823/447312=1.37001 

Are these differences within the range you would expect?

Show
Lauren MacArthur added a comment - - edited Great, thanks John! The logs are located on lsst-devl at: Gen2: / datasets / hsc / repo / rerun / RC / w_2022_04 / DM - 33457 / logs / jointcal / Gen3: / scratch / brendal4 / bps - gen3 - rc2 / w_2022_04 / submit / HSC / runs / RC2 / w_2022_04 / DM - 33402 / 20220128T212035Z / jobs / jointcal / The logs are not terribly easy to parse & compare, so as an example, I pulled out all the lines with chi2  &/or ndof  in them and compare the Gen2 vs. Gen3 versions directly. So, for / datasets / hsc / repo / rerun / RC / w_2022_04 / DM - 33457 / logs / jointcal / jointcal9813 - HSC - I - 59795.log vs. / scratch / brendal4 / bps - gen3 - rc2 / w_2022_04 / submit / HSC / runs / RC2 / w_2022_04 / DM - 33402 / 20220128T212035Z / jobs / jointcal / 9813 / HSC - I / e6e6d88e - f483 - 4041 - 98ef - 8390437230c7_jointcal_9813_ . 9754984.err the corresponding lines are: Gen2: Initial chi2 / ndof : 4.21833e + 16 / 490498 = 8.60009e + 10 Gen3: Initial chi2 / ndof : 4.21833e + 16 / 490498 = 8.60009e + 10   Gen2: Initialize DistortionsVisit chi2 / ndof : 1.87784e + 08 / 488122 = 384.707 Gen3: Initialize DistortionsVisit chi2 / ndof : 1.87614e + 08 / 488122 = 384.358   Gen2: Initialize Distortions chi2 / ndof : 9.44308e + 06 / 487510 = 19.37 Gen3: Initialize Distortions chi2 / ndof : 9.27373e + 06 / 487510 = 19.0227   Gen2: Initialize Positions chi2 / ndof : 1.96279e + 06 / 454112 = 4.32226 Gen3: Initialize Positions chi2 / ndof : 1.86372e + 06 / 454112 = 4.10409   Gen2: Initialize DistortionsPositions chi2 / ndof : 1.62282e + 06 / 451124 = 3.59729 Gen3: Initialize DistortionsPositions chi2 / ndof : 1.52377e + 06 / 451124 = 3.37773   Gen2: Iterations stopped with chi2 cut at 81.596 and relative change of - 0.0947015 Gen3: Iterations stopped with chi2 cut at 81.7825 and relative change of - 0.109253   Gen2: Fit iteration 0 chi2 / ndof : 803915 / 450176 = 1.78578 Gen3: Fit iteration 0 chi2 / ndof : 799876 / 450090 = 1.77715   Gen2: Iterations stopped with chi2 cut at 23.7586 and relative change of - 0.00395254 Gen3: Iterations stopped with chi2 cut at 23.9852 and relative change of - 0.00407391   Gen2: Fit completed chi2 / ndof : 611196 / 447240 = 1.36659 Gen3: Fit completed chi2 / ndof : 612823 / 447312 = 1.37001 Are these differences within the range you would expect?
Hide
John Parejko added a comment -

Differences in the initialization chi2/ndof are unsurprising (that's the ordering issue). Seeing that iterations stopped at a slightly different point during the fit I think is exactly what I'd expect (it's not following exactly the same gradient). If its stopped from fully converging, there will be some remaining difference. I don't really know what to "expect" here, but this doesn't seem unreasonable. Having of order 100 stars different (the final ndof value) is believable, I think.

Show
John Parejko added a comment - Differences in the initialization chi2/ndof are unsurprising (that's the ordering issue). Seeing that iterations stopped at a slightly different point during the fit I think is exactly what I'd expect (it's not following exactly the same gradient). If its stopped from fully converging, there will be some remaining difference. I don't really know what to "expect" here, but this doesn't seem unreasonable. Having of order 100 stars different (the final ndof value) is believable, I think.
Hide
Lauren MacArthur added a comment -

Great, thanks again!  I'll update the confluence page to confirm we agree that Gen2 support for jointcal can now be considered a thing of the past!

Show
Lauren MacArthur added a comment - Great, thanks again!  I'll update the  confluence page to confirm we agree that Gen2 support for jointcal can now be considered a thing of the past!

#### People

Assignee:
Lauren MacArthur
Reporter:
Lauren MacArthur
Reviewers:
John Parejko
Watchers:
Clare Saunders, John Parejko, Lauren MacArthur