Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-33760

Compare the jointcal SkyWcs's of the gen2 vs. gen3 w_2022_04 RC2 runs

    XMLWordPrintable

    Details

    • Story Points:
      2
    • Epic Link:
    • Sprint:
      DRP S22A
    • Team:
      Data Release Production
    • Urgent?:
      No

      Description

      As promised on DM-29821, this ticket is to compare the jointcal external SkyWcs objects from the HSC-RC2 w_2022_04 processing runs: Gen2 (DM-33457) vs. Gen3 (DM-33402). As noted here, the Gen2 run used a special-purpose config override to best match the current Gen3 configs. With the fixes on DM-29008 we are expecting very tight comparisons (with a likely small residual difference stemming from a difference in the order of inputs between Gen2 & Gen3).

        Attachments

          Issue Links

            Activity

            No builds found.
            lauren Lauren MacArthur created issue -
            lauren Lauren MacArthur made changes -
            Field Original Value New Value
            Link This issue relates to DM-29821 [ DM-29821 ]
            lauren Lauren MacArthur made changes -
            Link This issue relates to DM-33457 [ DM-33457 ]
            lauren Lauren MacArthur made changes -
            Link This issue relates to DM-33402 [ DM-33402 ]
            lauren Lauren MacArthur made changes -
            Link This issue relates to DM-29008 [ DM-29008 ]
            lauren Lauren MacArthur made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            lauren Lauren MacArthur made changes -
            lauren Lauren MacArthur made changes -
            lauren Lauren MacArthur made changes -
            lauren Lauren MacArthur made changes -
            lauren Lauren MacArthur made changes -
            lauren Lauren MacArthur made changes -
            lauren Lauren MacArthur made changes -
            lauren Lauren MacArthur made changes -
            Hide
            lauren Lauren MacArthur added a comment - - edited

            The speculations in the description indeed prove true.  The differences are down to levels so low (typically of order mean difference[stdev] ~0.001[0.02] mas for S/N>100 stars, and this is down by large factors of ~10-100 from the differences in the previous w_2021_46 runs) that I don't think they are of any scientific interest.  These remaining differences are almost certainly due to differences in the order of inputs and are not worth further investigation (noting that it will be up to the Gen3 version to provide the ability to order its inputs deterministically).  As such, I am ready to pull the plug on Gen2 support for jointcal.

            For a few more details, the following is our worst case in terms of the mean Gen2/Gen3 difference in Delta(RA)*cos(Dec) for S/N>100 stars (and note I did also look at the individual Delta(RA) and Delta(Dec) but did not notice any distinctions worth any scrutiny):



            This mean[stdev] difference is down from 2.80[1.15] mas in the previous w_2021_46 runs.

            We also see some that look to have some stochasticity in them (but do note the 1e-7 on the y-axis scale!):


            (note the stochasticity is not detector-specific, but rather they are spread throughout the focal plane).

            And a handful of intra-detector gradients:

            Overall, even our seemingly worst-cases are at levels so low as to be uninteresting.

            Show
            lauren Lauren MacArthur added a comment - - edited The speculations in the description indeed prove true.  The differences are down to levels so low (typically of order mean difference [stdev] ~0.001 [0.02]  mas for S/N>100 stars, and this is down by large factors of ~10-100 from the differences in the previous w_2021_46  runs) that I don't think they are of any scientific interest.  These remaining differences are almost certainly due to differences in the order of inputs and are not worth further investigation (noting that it will be up to the Gen3 version to provide the ability to order its inputs deterministically).  As such, I am ready to pull the plug on Gen2 support for jointcal . For a few more details, the following is our worst case in terms of the mean Gen2/Gen3 difference in Delta(RA)*cos(Dec) for S/N>100 stars (and note I did also look at the individual Delta(RA) and Delta(Dec) but did not notice any distinctions worth any scrutiny): This mean [stdev] difference is down from 2.80 [1.15] mas in the previous w_2021_46 runs. We also see some that look to have some stochasticity in them (but do note the 1e-7 on the y-axis scale!): (note the stochasticity is not detector-specific, but rather they are spread throughout the focal plane). And a handful of intra-detector gradients: Overall, even our seemingly worst-cases are at levels so low as to be uninteresting.
            Hide
            lauren Lauren MacArthur added a comment - - edited

            As far as how the above visit-level differences propagate into coadd differences, the effects are again very small (differences typically of order mean[stdev] ~0.003[0.06] mas for S/N~>500 stars, and again down by large factors of ~10 from the differences in the previous w_2021_46 runs)

            Here is our worst case scenario:


            and a more typical case:


            See DM-29820 if interested in the differences in the Stellar Locus plots.

            Again, this looks overall, to be small enough as to be uninteresting.

            Show
            lauren Lauren MacArthur added a comment - - edited As far as how the above visit-level differences propagate into coadd differences, the effects are again very small (differences typically of order mean [stdev] ~0.003 [0.06]  mas for S/N~>500 stars, and again down by large factors of ~10 from the differences in the previous w_2021_46  runs) Here is our worst case scenario: and a more typical case: See DM-29820 if interested in the differences in the Stellar Locus plots. Again, this looks overall, to be small enough as to be uninteresting.
            Hide
            lauren Lauren MacArthur added a comment -

            With the above results, I am prepared to pull the plug on Gen2 support for jointcal, but
            let me know what you think. If there are any particular cases you want looked at, let me know, or feel free to poke around yourself at:
            Gen2:
            https://lsst.ncsa.illinois.edu/~lauren/HSC_RC2/w_2022_04/plots/
            Gen3:
            https://lsst.ncsa.illinois.edu/~lauren/HSC_RC2/w_2022_04/vsGen3/plots/
            (it is the latter that has the direct comparison plots).

            Show
            lauren Lauren MacArthur added a comment - With the above results, I am prepared to pull the plug on Gen2 support for jointcal , but let me know what you think. If there are any particular cases you want looked at, let me know, or feel free to poke around yourself at: Gen2: https://lsst.ncsa.illinois.edu/~lauren/HSC_RC2/w_2022_04/plots/ Gen3: https://lsst.ncsa.illinois.edu/~lauren/HSC_RC2/w_2022_04/vsGen3/plots/ (it is the latter that has the direct comparison plots).
            lauren Lauren MacArthur made changes -
            Reviewers John Parejko [ parejkoj ]
            Status In Progress [ 3 ] In Review [ 10004 ]
            lauren Lauren MacArthur made changes -
            Remote Link This issue links to "Page (Confluence)" [ 32226 ]
            Parejkoj John Parejko made changes -
            Watchers John Parejko, Lauren MacArthur [ John Parejko, Lauren MacArthur ] Clare Saunders, John Parejko, Lauren MacArthur [ Clare Saunders, John Parejko, Lauren MacArthur ]
            Hide
            Parejkoj John Parejko added a comment -

            Jointcal input ordering is something I wondered about from early on (see DM-9393). It's a small effect, and I could believe the difference you see here is entirely due to that. When the stars are associated across visits, the exact details of the cross matches can be slightly different, but that difference should come out by the end of the fit. Can you give me links to the logs for a couple of matched gen2/gen3 runs from this comparison? I'm curious whether the final number of stars in the fit and the final chi2 and ndof values are the same or slightly different? If the fitting process is perfect, I'd expect them to be identical (any ambiguous matches would have been removed), but HSC is now using astrometryOutlierRelativeTolerance, which means the fit ends before it's fully converged (to reduce the runtime). I'd bet that that accounts for most (all?) of the difference you see here. I don't think it's worth exploring what gen2/gen3 differences on HSC data look like with that config set to None, since it's so small, but a quick check of the final gen2/gen3 chi2 and ndof, values might be informative, as to how different jointcal thinks it is in those two cases.

            This also reminds me that jointcal doesn't include a final count of the number of fittedStars/measuredStars. Those get incorporated into ndof, so it's probably not relevant most of the time.

            Show
            Parejkoj John Parejko added a comment - Jointcal input ordering is something I wondered about from early on (see DM-9393 ). It's a small effect, and I could believe the difference you see here is entirely due to that. When the stars are associated across visits, the exact details of the cross matches can be slightly different, but that difference should come out by the end of the fit. Can you give me links to the logs for a couple of matched gen2/gen3 runs from this comparison? I'm curious whether the final number of stars in the fit and the final chi2 and ndof values are the same or slightly different? If the fitting process is perfect, I'd expect them to be identical (any ambiguous matches would have been removed), but HSC is now using astrometryOutlierRelativeTolerance , which means the fit ends before it's fully converged (to reduce the runtime). I'd bet that that accounts for most (all?) of the difference you see here. I don't think it's worth exploring what gen2/gen3 differences on HSC data look like with that config set to None, since it's so small, but a quick check of the final gen2/gen3 chi2 and ndof, values might be informative, as to how different jointcal thinks it is in those two cases. This also reminds me that jointcal doesn't include a final count of the number of fittedStars/measuredStars. Those get incorporated into ndof, so it's probably not relevant most of the time.
            Parejkoj John Parejko made changes -
            Status In Review [ 10004 ] Reviewed [ 10101 ]
            Hide
            lauren Lauren MacArthur added a comment - - edited

            Great, thanks John!

            The logs are located on lsst-devl at:

             Gen2: /datasets/hsc/repo/rerun/RC/w_2022_04/DM-33457/logs/jointcal/
             Gen3: /scratch/brendal4/bps-gen3-rc2/w_2022_04/submit/HSC/runs/RC2/w_2022_04/DM-33402/20220128T212035Z/jobs/jointcal/
            

            The logs are not terribly easy to parse & compare, so as an example, I pulled out all the lines with chi2 &/or ndof in them and compare the Gen2 vs. Gen3 versions directly. So, for

            /datasets/hsc/repo/rerun/RC/w_2022_04/DM-33457/logs/jointcal/jointcal9813-HSC-I-59795.log
            

            vs.

            /scratch/brendal4/bps-gen3-rc2/w_2022_04/submit/HSC/runs/RC2/w_2022_04/DM-33402/20220128T212035Z/jobs/jointcal/9813/HSC-I/e6e6d88e-f483-4041-98ef-8390437230c7_jointcal_9813_.9754984.err 
            

            the corresponding lines are:

            Gen2: Initial chi2/ndof : 4.21833e+16/490498=8.60009e+10
            Gen3: Initial chi2/ndof : 4.21833e+16/490498=8.60009e+10
             
            Gen2: Initialize DistortionsVisit chi2/ndof : 1.87784e+08/488122=384.707
            Gen3: Initialize DistortionsVisit chi2/ndof : 1.87614e+08/488122=384.358
             
            Gen2: Initialize Distortions chi2/ndof : 9.44308e+06/487510=19.37
            Gen3: Initialize Distortions chi2/ndof : 9.27373e+06/487510=19.0227
             
            Gen2: Initialize Positions chi2/ndof : 1.96279e+06/454112=4.32226
            Gen3: Initialize Positions chi2/ndof : 1.86372e+06/454112=4.10409
             
            Gen2: Initialize DistortionsPositions chi2/ndof : 1.62282e+06/451124=3.59729
            Gen3: Initialize DistortionsPositions chi2/ndof : 1.52377e+06/451124=3.37773
             
            Gen2: Iterations stopped with chi2 cut at 81.596 and relative change of -0.0947015
            Gen3: Iterations stopped with chi2 cut at 81.7825 and relative change of -0.109253
             
            Gen2: Fit iteration 0 chi2/ndof : 803915/450176=1.78578
            Gen3: Fit iteration 0 chi2/ndof : 799876/450090=1.77715
             
            Gen2: Iterations stopped with chi2 cut at 23.7586 and relative change of -0.00395254
            Gen3: Iterations stopped with chi2 cut at 23.9852 and relative change of -0.00407391
             
            Gen2: Fit completed chi2/ndof : 611196/447240=1.36659
            Gen3: Fit completed chi2/ndof : 612823/447312=1.37001
            

            Are these differences within the range you would expect?

            Show
            lauren Lauren MacArthur added a comment - - edited Great, thanks John! The logs are located on lsst-devl at: Gen2: / datasets / hsc / repo / rerun / RC / w_2022_04 / DM - 33457 / logs / jointcal / Gen3: / scratch / brendal4 / bps - gen3 - rc2 / w_2022_04 / submit / HSC / runs / RC2 / w_2022_04 / DM - 33402 / 20220128T212035Z / jobs / jointcal / The logs are not terribly easy to parse & compare, so as an example, I pulled out all the lines with chi2  &/or ndof  in them and compare the Gen2 vs. Gen3 versions directly. So, for / datasets / hsc / repo / rerun / RC / w_2022_04 / DM - 33457 / logs / jointcal / jointcal9813 - HSC - I - 59795.log vs. / scratch / brendal4 / bps - gen3 - rc2 / w_2022_04 / submit / HSC / runs / RC2 / w_2022_04 / DM - 33402 / 20220128T212035Z / jobs / jointcal / 9813 / HSC - I / e6e6d88e - f483 - 4041 - 98ef - 8390437230c7_jointcal_9813_ . 9754984.err the corresponding lines are: Gen2: Initial chi2 / ndof : 4.21833e + 16 / 490498 = 8.60009e + 10 Gen3: Initial chi2 / ndof : 4.21833e + 16 / 490498 = 8.60009e + 10   Gen2: Initialize DistortionsVisit chi2 / ndof : 1.87784e + 08 / 488122 = 384.707 Gen3: Initialize DistortionsVisit chi2 / ndof : 1.87614e + 08 / 488122 = 384.358   Gen2: Initialize Distortions chi2 / ndof : 9.44308e + 06 / 487510 = 19.37 Gen3: Initialize Distortions chi2 / ndof : 9.27373e + 06 / 487510 = 19.0227   Gen2: Initialize Positions chi2 / ndof : 1.96279e + 06 / 454112 = 4.32226 Gen3: Initialize Positions chi2 / ndof : 1.86372e + 06 / 454112 = 4.10409   Gen2: Initialize DistortionsPositions chi2 / ndof : 1.62282e + 06 / 451124 = 3.59729 Gen3: Initialize DistortionsPositions chi2 / ndof : 1.52377e + 06 / 451124 = 3.37773   Gen2: Iterations stopped with chi2 cut at 81.596 and relative change of - 0.0947015 Gen3: Iterations stopped with chi2 cut at 81.7825 and relative change of - 0.109253   Gen2: Fit iteration 0 chi2 / ndof : 803915 / 450176 = 1.78578 Gen3: Fit iteration 0 chi2 / ndof : 799876 / 450090 = 1.77715   Gen2: Iterations stopped with chi2 cut at 23.7586 and relative change of - 0.00395254 Gen3: Iterations stopped with chi2 cut at 23.9852 and relative change of - 0.00407391   Gen2: Fit completed chi2 / ndof : 611196 / 447240 = 1.36659 Gen3: Fit completed chi2 / ndof : 612823 / 447312 = 1.37001 Are these differences within the range you would expect?
            Hide
            Parejkoj John Parejko added a comment -

            Differences in the initialization chi2/ndof are unsurprising (that's the ordering issue). Seeing that iterations stopped at a slightly different point during the fit I think is exactly what I'd expect (it's not following exactly the same gradient). If its stopped from fully converging, there will be some remaining difference. I don't really know what to "expect" here, but this doesn't seem unreasonable. Having of order 100 stars different (the final ndof value) is believable, I think.

            Show
            Parejkoj John Parejko added a comment - Differences in the initialization chi2/ndof are unsurprising (that's the ordering issue). Seeing that iterations stopped at a slightly different point during the fit I think is exactly what I'd expect (it's not following exactly the same gradient). If its stopped from fully converging, there will be some remaining difference. I don't really know what to "expect" here, but this doesn't seem unreasonable. Having of order 100 stars different (the final ndof value) is believable, I think.
            Hide
            lauren Lauren MacArthur added a comment -

            Great, thanks again!  I'll update the confluence page to confirm we agree that Gen2 support for jointcal can now be considered a thing of the past!

            Show
            lauren Lauren MacArthur added a comment - Great, thanks again!  I'll update the  confluence page to confirm we agree that Gen2 support for jointcal can now be considered a thing of the past!
            lauren Lauren MacArthur made changes -
            Resolution Done [ 10000 ]
            Status Reviewed [ 10101 ] Done [ 10002 ]
            lauren Lauren MacArthur made changes -
            Story Points 2

              People

              Assignee:
              lauren Lauren MacArthur
              Reporter:
              lauren Lauren MacArthur
              Reviewers:
              John Parejko
              Watchers:
              Clare Saunders, John Parejko, Lauren MacArthur
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.