Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-18630

Investigate jointcal astrometry models with Run2.1i data

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Story Points:
      8
    • Epic Link:
    • Sprint:
      DRP S19-4, DRP S19-5
    • Team:
      Data Release Production

      Description

      DESC reported that running jointcal on Run2.1i made the astrometry repeatability worse than processCcd.  This is not surprising because:

      1) The reference catalogs are near perfect and to full depth < 23.  They essentially are the input catalogs, and therefor  jointcal's benefit of bootstrapping deeper is not realized. 

      2) ProcessCcd fits a 3rd order polynomial per-ccd. Jointcal is fitting a model that is 1st order per ccd and 7th order per focal plane. The jointcal model is more robust, but doesn't have as many degrees of freedom as the processCcd model. 

      James Chiang copied over a tract to lsst-dev for us to play with. Thanks to Hsin-Fang Chiang for ingesting it and running it through singleFrame processing. On this ticket, I'll post the results of:

      1) Running validateDrp and visitAnalysis on 4 different config repos:

      2) Reconciling metrics posted in validateDrp and visitAnalysis from the ones computed in Jim's notebook

      3) Rerunning single frame using only bright stars from the reference catalog. 

        Attachments

          Issue Links

            Activity

            Hide
            yusra Yusra AlSayyad added a comment -

            Update on step one posted in this notebook.
            TL;DR. Jointcal does perform better when it has same or more orders. Repeatability < 5 mas.

            https://github.com/yalsayyad/dm_notebooks/blob/master/calibration/run21i_validate_drp.ipynb

            on to steps #2 and #3.

            Show
            yusra Yusra AlSayyad added a comment - Update on step one posted in this notebook. TL;DR. Jointcal does perform better when it has same or more orders. Repeatability < 5 mas. https://github.com/yalsayyad/dm_notebooks/blob/master/calibration/run21i_validate_drp.ipynb on to steps #2 and #3.
            Hide
            yusra Yusra AlSayyad added a comment - - edited

            Updated the notebook to include the takeaway from the `visitAnalysis` plots, and github struggles to render it now. Try:
            https://nbviewer.jupyter.org/github/yalsayyad/dm_notebooks/blob/master/calibration/run21i_validate_drp.ipynb

            TL;DR: Master-branch jointcal reproduces the reference catalog positions worse than processCcd because the assumed reference catalog position errors are too high (100mas). If this is reduced to 1mas, we get both better repeatability and matching with the reference catalog. John Parejko referred me to ticket DM-10826.

            Show
            yusra Yusra AlSayyad added a comment - - edited Updated the notebook to include the takeaway from the ` visitAnalysis ` plots, and github struggles to render it now. Try: https://nbviewer.jupyter.org/github/yalsayyad/dm_notebooks/blob/master/calibration/run21i_validate_drp.ipynb TL;DR: Master-branch jointcal reproduces the reference catalog positions worse than processCcd because the assumed reference catalog position errors are too high (100mas). If this is reduced to 1mas, we get both better repeatability and matching with the reference catalog. John Parejko referred me to ticket DM-10826 .
            Hide
            yusra Yusra AlSayyad added a comment -

            Step 2) Reconciling RefCat - Output metrics posted in visitAnalysis from the ones computed in James Chiang's notebook

            TL;DR: Notebook and visitAnalysis are consistent in the sense that distance to the reference catalog both sets of plots show that: jointcal_default (worst) > processCcd > jointcal_simple_3_1mas (best).
            At the moment, visitAnalysis doesn't report a median distance to reference catalog (with jointcal calibs), but does report dec - dec_ref and RaCosDec - RaCosDec_ref, vs. mag and skymap.

            I reran your notrbook keeping 'slot_modelFlux' for two of the reruns to represent a "before" and an "after":
            before = jointcal_default
            after = jointcal_simple_3_1mas

            For the jointcal_default run:
            https://lsst-web.ncsa.illinois.edu/~yusra/QA_DC2/JimPlots/Run2.1i_i-band_7_visits_astrometry_constrained.png
            https://lsst-web.ncsa.illinois.edu/~yusra/QA_DC2/JimPlots/Run2.1i_i-band_7_visits_astrometry_default_20.pngAlso binned density while I had the data loaded:
            https://lsst-web.ncsa.illinois.edu/~yusra/QA_DC2/JimPlots/Run2.1i_i-band_7_visits_astrometry_density_default_21.png

            For the jointcal_simple_3_1mas run:
            https://lsst-web.ncsa.illinois.edu/~yusra/QA_DC2/JimPlots/Run2.1i_i-band_7_visits_astrometry_simple3_1mas_4panel.png
            Also binned density while I had the data loaded:
            https://lsst-web.ncsa.illinois.edu/~yusra/QA_DC2/JimPlots/Run2.1i_i-band_7_visits_astrometry_density_default_simple_3_1mas_21.png

            Summary numbers from Jim's notebook
            6.2mas processCcd, 10.0mas jointcal_default, and 5.0mas simple_3_1mas for i<21 (Jim's Notebook)
            5.4mas processCcd, 9.4mas joincal_default, and 4.2mas simple_3_1mas for i<20 (Jim's Notebook)

            Show
            yusra Yusra AlSayyad added a comment - Step 2) Reconciling RefCat - Output metrics posted in visitAnalysis from the ones computed in James Chiang 's notebook TL;DR: Notebook and visitAnalysis are consistent in the sense that distance to the reference catalog both sets of plots show that: jointcal_default (worst) > processCcd > jointcal_simple_3_1mas (best). At the moment, visitAnalysis doesn't report a median distance to reference catalog (with jointcal calibs), but does report dec - dec_ref and RaCosDec - RaCosDec_ref, vs. mag and skymap. I reran your notrbook keeping 'slot_modelFlux' for two of the reruns to represent a "before" and an "after": before = jointcal_default after = jointcal_simple_3_1mas For the jointcal_default run: https://lsst-web.ncsa.illinois.edu/~yusra/QA_DC2/JimPlots/Run2.1i_i-band_7_visits_astrometry_constrained.png https://lsst-web.ncsa.illinois.edu/~yusra/QA_DC2/JimPlots/Run2.1i_i-band_7_visits_astrometry_default_20.png Also binned density while I had the data loaded: https://lsst-web.ncsa.illinois.edu/~yusra/QA_DC2/JimPlots/Run2.1i_i-band_7_visits_astrometry_density_default_21.png For the jointcal_simple_3_1mas run: https://lsst-web.ncsa.illinois.edu/~yusra/QA_DC2/JimPlots/Run2.1i_i-band_7_visits_astrometry_simple3_1mas_4panel.png Also binned density while I had the data loaded: https://lsst-web.ncsa.illinois.edu/~yusra/QA_DC2/JimPlots/Run2.1i_i-band_7_visits_astrometry_density_default_simple_3_1mas_21.png Summary numbers from Jim's notebook 6.2mas processCcd, 10.0mas jointcal_default, and 5.0mas simple_3_1mas for i<21 (Jim's Notebook) 5.4mas processCcd, 9.4mas joincal_default, and 4.2mas simple_3_1mas for i<20 (Jim's Notebook)
            Hide
            yusra Yusra AlSayyad added a comment -

            Marking this done with 2 of the 3 tasks completed. The third is postponed for a new ticket: 
            DM-19503: Use DC2 Run2.1i to show how jointcal improves astrom vs ref cat flux cutoff

            Conclusions from this ticket:

            • We showed that jointcal does perform as expected (slightly better) when switching the configs to use a model representative of the simulation's model.
            • We reproduced James Chiang results (difference between input and output position) which turned out to be due to the assumed refcat errors of 100mas. This resulted in the prioritization of existing ticket DM-10826 .
            Show
            yusra Yusra AlSayyad added a comment - Marking this done with 2 of the 3 tasks completed. The third is postponed for a new ticket:  DM-19503 : Use DC2 Run2.1i to show how jointcal improves astrom vs ref cat flux cutoff Conclusions from this ticket: We showed that jointcal does perform as expected (slightly better) when switching the configs to use a model representative of the simulation's model. We reproduced James Chiang results (difference between input and output position) which turned out to be due to the assumed refcat errors of 100mas. This resulted in the prioritization of existing ticket DM-10826 .

              People

              Assignee:
              yusra Yusra AlSayyad
              Reporter:
              yusra Yusra AlSayyad
              Watchers:
              Eli Rykoff, James Chiang, John Parejko, Yusra AlSayyad
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.