Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-25132

Calculate parameter covariances from Gaia correlations

    XMLWordPrintable

    Details

    • Type: Story
    • Status: In Review
    • Resolution: Unresolved
    • Fix Version/s: None
    • Component/s: meas_astrom
    • Labels:
      None
    • Team:
      Alert Production
    • Urgent?:
      No

      Description

      In a video chat, Pierre Astier pointed out to me that the Gaia DR2 catalog provides position parameter correlations, which can be quite significant for many sources, sometimes even larger than the single parameter errors. We have slots in our refcats for covariances, but no code to do the conversion nor config options to extract those values from an external catalog (see note in the IngestIndexedReferenceTask class docstring). Now that we have Gaia DR2 testdata for jointcal to process, I noticed that the final chi2 were very low (~1e-3), which may be related to this issue.

      At minimum, if we can convert the ra/dec correlations to covariances (Pierre Astier said a colleague had code for that), jointcal should be able to use them relatively easily. Making use of the other position covariances (e.g. ra_parallax_corr, dec_pmra_corr) would require additional code in jointcal.

      How much this affects jointcal's final fit parameters vs. just the fit statistics is an open question. We might want to just wait for Gaia DR3 to be released (originally planned for mid-2021, but now delayed due to COVID-19), as it should have reduced correlations and may have a different format.

      Here's the link to the DR2 GAIA_SOURCE catalog datamodel, since I always have to click through a bunch of pages to find it:
      https://gea.esac.esa.int/archive/documentation/GDR3/Gaia_archive/chap_datamodel/sec_dm_main_source_catalogue/ssec_dm_gaia_source.html

        Attachments

          Issue Links

            Activity

            Hide
            Parejkoj John Parejko added a comment -

            Note for future me, since I just figured this out: Gaia correlation coefficients are rho_xy such that the off-diagonal element of the covariance matrix is:

            V_ij = sigma_x*sigma_y*rho_xy
            

            where x,y are each of the relevant parameters. So, computing the relevant covariances should be pretty simple in IngestGaiaManager given the above relationship. However, we probably also have to change makeSchema and the ingest Config to increase the dimensions of the respective errors. makeMinimalSchema also would need an update to support e.g. ra_pm covariances.

            See "Error propagation" here:

            https://gea.esac.esa.int/archive/documentation/GDR2/Data_processing/chap_cu3ast/sec_cu3ast_intro/ssec_cu3ast_intro_tansforms.html

            Show
            Parejkoj John Parejko added a comment - Note for future me, since I just figured this out: Gaia correlation coefficients are rho_xy such that the off-diagonal element of the covariance matrix is: V_ij = sigma_x*sigma_y*rho_xy where x,y are each of the relevant parameters. So, computing the relevant covariances should be pretty simple in IngestGaiaManager given the above relationship. However, we probably also have to change makeSchema and the ingest Config to increase the dimensions of the respective errors. makeMinimalSchema also would need an update to support e.g. ra_pm covariances. See "Error propagation" here: https://gea.esac.esa.int/archive/documentation/GDR2/Data_processing/chap_cu3ast/sec_cu3ast_intro/ssec_cu3ast_intro_tansforms.html
            Hide
            Parejkoj John Parejko added a comment - - edited

            We now have makeMinimalSchema setup to support the full covariance matrix, with some tests of it. I've deprecated properMotionErrDim, since we'll either have the full matrix, or no PM errors.

            Next steps:

            • Fix the errors in TestConvertReferenceCatalogParallel and ReferenceObjectLoaderLoadTests.setUpClass related to withPmErr.
            • Fix how existing errors and covariances (if any?) are handled in ConvertRefcatManager.
            • Add support for covariances to ConvertRefcatManager; not sure if this is actually necessary, or if everything should go into the Gaia-specific subclass?
            • Add Gaia correlation->covariance math to ConvertGaiaManager.
            Show
            Parejkoj John Parejko added a comment - - edited We now have makeMinimalSchema setup to support the full covariance matrix, with some tests of it. I've deprecated properMotionErrDim , since we'll either have the full matrix, or no PM errors. Next steps: Fix the errors in TestConvertReferenceCatalogParallel and ReferenceObjectLoaderLoadTests.setUpClass related to withPmErr . Fix how existing errors and covariances (if any?) are handled in ConvertRefcatManager . Add support for covariances to ConvertRefcatManager ; not sure if this is actually necessary, or if everything should go into the Gaia-specific subclass? Add Gaia correlation->covariance math to ConvertGaiaManager .
            Show
            csaunder Clare Saunders added a comment - Successful Jenkins run:  https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/37599/pipeline .
            Hide
            krzys Krzysztof Findeisen added a comment -

            Sorry for taking so long with the review! Unfortunately, while it's nice to see proper astrometric error-handling in the stack, a lot of the changes seem to be working at cross-purposes with each other or with backward-compatibility. I'm happy to re-review once those have been addressed.

            Show
            krzys Krzysztof Findeisen added a comment - Sorry for taking so long with the review! Unfortunately, while it's nice to see proper astrometric error-handling in the stack, a lot of the changes seem to be working at cross-purposes with each other or with backward-compatibility. I'm happy to re-review once those have been addressed.
            Hide
            csaunder Clare Saunders added a comment -

            Don't worry–it's been less than 48 hours!

            I understand what you are saying about the lack of coherence of the changes. I think we made some design choices at the beginning that led to things being overly complicated. I'm going to think about whether I can refactor the changes so that they're more straightforward.

            Show
            csaunder Clare Saunders added a comment - Don't worry–it's been less than 48 hours! I understand what you are saying about the lack of coherence of the changes. I think we made some design choices at the beginning that led to things being overly complicated. I'm going to think about whether I can refactor the changes so that they're more straightforward.

              People

              Assignee:
              csaunder Clare Saunders
              Reporter:
              Parejkoj John Parejko
              Reviewers:
              Krzysztof Findeisen
              Watchers:
              Clare Saunders, Eric Bellm, Jim Bosch, John Parejko, John Swinbank, Krzysztof Findeisen, Pierre Astier
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

                Dates

                Created:
                Updated:

                  Jenkins Builds

                  No builds found.