Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-18630

Investigate jointcal astrometry models with Run2.1i data

    XMLWordPrintable

Details

    • Story
    • Status: Done
    • Resolution: Done
    • None
    • None
    • None
    • 8
    • DRP S19-4, DRP S19-5
    • Data Release Production

    Description

      DESC reported that running jointcal on Run2.1i made the astrometry repeatability worse than processCcd.  This is not surprising because:

      1) The reference catalogs are near perfect and to full depth < 23.  They essentially are the input catalogs, and therefor  jointcal's benefit of bootstrapping deeper is not realized. 

      2) ProcessCcd fits a 3rd order polynomial per-ccd. Jointcal is fitting a model that is 1st order per ccd and 7th order per focal plane. The jointcal model is more robust, but doesn't have as many degrees of freedom as the processCcd model. 

      jchiang copied over a tract to lsst-dev for us to play with. Thanks to hchiang2 for ingesting it and running it through singleFrame processing. On this ticket, I'll post the results of:

      1) Running validateDrp and visitAnalysis on 4 different config repos:

      2) Reconciling metrics posted in validateDrp and visitAnalysis from the ones computed in Jim's notebook

      3) Rerunning single frame using only bright stars from the reference catalog. 

      Attachments

        Issue Links

          Activity

            Update on step one posted in this notebook.
            TL;DR. Jointcal does perform better when it has same or more orders. Repeatability < 5 mas.

            https://github.com/yalsayyad/dm_notebooks/blob/master/calibration/run21i_validate_drp.ipynb

            on to steps #2 and #3.

            yusra Yusra AlSayyad added a comment - Update on step one posted in this notebook. TL;DR. Jointcal does perform better when it has same or more orders. Repeatability < 5 mas. https://github.com/yalsayyad/dm_notebooks/blob/master/calibration/run21i_validate_drp.ipynb on to steps #2 and #3.
            yusra Yusra AlSayyad added a comment - - edited

            Updated the notebook to include the takeaway from the `visitAnalysis` plots, and github struggles to render it now. Try:
            https://nbviewer.jupyter.org/github/yalsayyad/dm_notebooks/blob/master/calibration/run21i_validate_drp.ipynb

            TL;DR: Master-branch jointcal reproduces the reference catalog positions worse than processCcd because the assumed reference catalog position errors are too high (100mas). If this is reduced to 1mas, we get both better repeatability and matching with the reference catalog. Parejkoj referred me to ticket DM-10826.

            yusra Yusra AlSayyad added a comment - - edited Updated the notebook to include the takeaway from the ` visitAnalysis ` plots, and github struggles to render it now. Try: https://nbviewer.jupyter.org/github/yalsayyad/dm_notebooks/blob/master/calibration/run21i_validate_drp.ipynb TL;DR: Master-branch jointcal reproduces the reference catalog positions worse than processCcd because the assumed reference catalog position errors are too high (100mas). If this is reduced to 1mas, we get both better repeatability and matching with the reference catalog. Parejkoj referred me to ticket DM-10826 .

            Step 2) Reconciling RefCat - Output metrics posted in visitAnalysis from the ones computed in jchiang's notebook

            TL;DR: Notebook and visitAnalysis are consistent in the sense that distance to the reference catalog both sets of plots show that: jointcal_default (worst) > processCcd > jointcal_simple_3_1mas (best).
            At the moment, visitAnalysis doesn't report a median distance to reference catalog (with jointcal calibs), but does report dec - dec_ref and RaCosDec - RaCosDec_ref, vs. mag and skymap.

            I reran your notrbook keeping 'slot_modelFlux' for two of the reruns to represent a "before" and an "after":
            before = jointcal_default
            after = jointcal_simple_3_1mas

            For the jointcal_default run:
            https://lsst-web.ncsa.illinois.edu/~yusra/QA_DC2/JimPlots/Run2.1i_i-band_7_visits_astrometry_constrained.png
            https://lsst-web.ncsa.illinois.edu/~yusra/QA_DC2/JimPlots/Run2.1i_i-band_7_visits_astrometry_default_20.pngAlso binned density while I had the data loaded:
            https://lsst-web.ncsa.illinois.edu/~yusra/QA_DC2/JimPlots/Run2.1i_i-band_7_visits_astrometry_density_default_21.png

            For the jointcal_simple_3_1mas run:
            https://lsst-web.ncsa.illinois.edu/~yusra/QA_DC2/JimPlots/Run2.1i_i-band_7_visits_astrometry_simple3_1mas_4panel.png
            Also binned density while I had the data loaded:
            https://lsst-web.ncsa.illinois.edu/~yusra/QA_DC2/JimPlots/Run2.1i_i-band_7_visits_astrometry_density_default_simple_3_1mas_21.png

            Summary numbers from Jim's notebook
            6.2mas processCcd, 10.0mas jointcal_default, and 5.0mas simple_3_1mas for i<21 (Jim's Notebook)
            5.4mas processCcd, 9.4mas joincal_default, and 4.2mas simple_3_1mas for i<20 (Jim's Notebook)

            yusra Yusra AlSayyad added a comment - Step 2) Reconciling RefCat - Output metrics posted in visitAnalysis from the ones computed in jchiang 's notebook TL;DR: Notebook and visitAnalysis are consistent in the sense that distance to the reference catalog both sets of plots show that: jointcal_default (worst) > processCcd > jointcal_simple_3_1mas (best). At the moment, visitAnalysis doesn't report a median distance to reference catalog (with jointcal calibs), but does report dec - dec_ref and RaCosDec - RaCosDec_ref, vs. mag and skymap. I reran your notrbook keeping 'slot_modelFlux' for two of the reruns to represent a "before" and an "after": before = jointcal_default after = jointcal_simple_3_1mas For the jointcal_default run: https://lsst-web.ncsa.illinois.edu/~yusra/QA_DC2/JimPlots/Run2.1i_i-band_7_visits_astrometry_constrained.png https://lsst-web.ncsa.illinois.edu/~yusra/QA_DC2/JimPlots/Run2.1i_i-band_7_visits_astrometry_default_20.png Also binned density while I had the data loaded: https://lsst-web.ncsa.illinois.edu/~yusra/QA_DC2/JimPlots/Run2.1i_i-band_7_visits_astrometry_density_default_21.png For the jointcal_simple_3_1mas run: https://lsst-web.ncsa.illinois.edu/~yusra/QA_DC2/JimPlots/Run2.1i_i-band_7_visits_astrometry_simple3_1mas_4panel.png Also binned density while I had the data loaded: https://lsst-web.ncsa.illinois.edu/~yusra/QA_DC2/JimPlots/Run2.1i_i-band_7_visits_astrometry_density_default_simple_3_1mas_21.png Summary numbers from Jim's notebook 6.2mas processCcd, 10.0mas jointcal_default, and 5.0mas simple_3_1mas for i<21 (Jim's Notebook) 5.4mas processCcd, 9.4mas joincal_default, and 4.2mas simple_3_1mas for i<20 (Jim's Notebook)

            Marking this done with 2 of the 3 tasks completed. The third is postponed for a new ticket: 
            DM-19503: Use DC2 Run2.1i to show how jointcal improves astrom vs ref cat flux cutoff

            Conclusions from this ticket:

            • We showed that jointcal does perform as expected (slightly better) when switching the configs to use a model representative of the simulation's model.
            • We reproduced jchiang results (difference between input and output position) which turned out to be due to the assumed refcat errors of 100mas. This resulted in the prioritization of existing ticket DM-10826 .
            yusra Yusra AlSayyad added a comment - Marking this done with 2 of the 3 tasks completed. The third is postponed for a new ticket:  DM-19503 : Use DC2 Run2.1i to show how jointcal improves astrom vs ref cat flux cutoff Conclusions from this ticket: We showed that jointcal does perform as expected (slightly better) when switching the configs to use a model representative of the simulation's model. We reproduced jchiang results (difference between input and output position) which turned out to be due to the assumed refcat errors of 100mas. This resulted in the prioritization of existing ticket DM-10826 .

            People

              yusra Yusra AlSayyad
              yusra Yusra AlSayyad
              Eli Rykoff, James Chiang, John Parejko, Yusra AlSayyad
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.