# Calculate parameter covariances from Gaia correlations

XMLWordPrintable

#### Details

• Type: Story
• Status: In Review
• Resolution: Unresolved
• Fix Version/s: None
• Component/s:
• Labels:
None
• Team:
• Urgent?:
No

#### Description

In a video chat, Pierre Astier pointed out to me that the Gaia DR2 catalog provides position parameter correlations, which can be quite significant for many sources, sometimes even larger than the single parameter errors. We have slots in our refcats for covariances, but no code to do the conversion nor config options to extract those values from an external catalog (see note in the IngestIndexedReferenceTask class docstring). Now that we have Gaia DR2 testdata for jointcal to process, I noticed that the final chi2 were very low (~1e-3), which may be related to this issue.

At minimum, if we can convert the ra/dec correlations to covariances (Pierre Astier said a colleague had code for that), jointcal should be able to use them relatively easily. Making use of the other position covariances (e.g. ra_parallax_corr, dec_pmra_corr) would require additional code in jointcal.

How much this affects jointcal's final fit parameters vs. just the fit statistics is an open question. We might want to just wait for Gaia DR3 to be released (originally planned for mid-2021, but now delayed due to COVID-19), as it should have reduced correlations and may have a different format.

Here's the link to the DR2 GAIA_SOURCE catalog datamodel, since I always have to click through a bunch of pages to find it:
https://gea.esac.esa.int/archive/documentation/GDR3/Gaia_archive/chap_datamodel/sec_dm_main_source_catalogue/ssec_dm_gaia_source.html

#### Activity

Hide
John Parejko added a comment -

Note for future me, since I just figured this out: Gaia correlation coefficients are rho_xy such that the off-diagonal element of the covariance matrix is:

 V_ij = sigma_x*sigma_y*rho_xy 

where x,y are each of the relevant parameters. So, computing the relevant covariances should be pretty simple in IngestGaiaManager given the above relationship. However, we probably also have to change makeSchema and the ingest Config to increase the dimensions of the respective errors. makeMinimalSchema also would need an update to support e.g. ra_pm covariances.

See "Error propagation" here:

https://gea.esac.esa.int/archive/documentation/GDR2/Data_processing/chap_cu3ast/sec_cu3ast_intro/ssec_cu3ast_intro_tansforms.html

Show
John Parejko added a comment - Note for future me, since I just figured this out: Gaia correlation coefficients are rho_xy such that the off-diagonal element of the covariance matrix is: V_ij = sigma_x*sigma_y*rho_xy where x,y are each of the relevant parameters. So, computing the relevant covariances should be pretty simple in IngestGaiaManager given the above relationship. However, we probably also have to change makeSchema and the ingest Config to increase the dimensions of the respective errors. makeMinimalSchema also would need an update to support e.g. ra_pm covariances. See "Error propagation" here: https://gea.esac.esa.int/archive/documentation/GDR2/Data_processing/chap_cu3ast/sec_cu3ast_intro/ssec_cu3ast_intro_tansforms.html
Hide
John Parejko added a comment - - edited

We now have makeMinimalSchema setup to support the full covariance matrix, with some tests of it. I've deprecated properMotionErrDim, since we'll either have the full matrix, or no PM errors.

Next steps:

• Fix how existing errors and covariances (if any?) are handled in ConvertRefcatManager.
• Add support for covariances to ConvertRefcatManager; not sure if this is actually necessary, or if everything should go into the Gaia-specific subclass?
• Add Gaia correlation->covariance math to ConvertGaiaManager.
Show
John Parejko added a comment - - edited We now have makeMinimalSchema setup to support the full covariance matrix, with some tests of it. I've deprecated properMotionErrDim , since we'll either have the full matrix, or no PM errors. Next steps: Fix the errors in TestConvertReferenceCatalogParallel and ReferenceObjectLoaderLoadTests.setUpClass related to withPmErr . Fix how existing errors and covariances (if any?) are handled in ConvertRefcatManager . Add support for covariances to ConvertRefcatManager ; not sure if this is actually necessary, or if everything should go into the Gaia-specific subclass? Add Gaia correlation->covariance math to ConvertGaiaManager .
Hide
Clare Saunders added a comment -
Show
Clare Saunders added a comment - Successful Jenkins run:  https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/37599/pipeline .
Hide
Krzysztof Findeisen added a comment -

Sorry for taking so long with the review! Unfortunately, while it's nice to see proper astrometric error-handling in the stack, a lot of the changes seem to be working at cross-purposes with each other or with backward-compatibility. I'm happy to re-review once those have been addressed.

Show
Krzysztof Findeisen added a comment - Sorry for taking so long with the review! Unfortunately, while it's nice to see proper astrometric error-handling in the stack, a lot of the changes seem to be working at cross-purposes with each other or with backward-compatibility. I'm happy to re-review once those have been addressed.
Hide
Clare Saunders added a comment -

Don't worry–it's been less than 48 hours!

I understand what you are saying about the lack of coherence of the changes. I think we made some design choices at the beginning that led to things being overly complicated. I'm going to think about whether I can refactor the changes so that they're more straightforward.

Show
Clare Saunders added a comment - Don't worry–it's been less than 48 hours! I understand what you are saying about the lack of coherence of the changes. I think we made some design choices at the beginning that led to things being overly complicated. I'm going to think about whether I can refactor the changes so that they're more straightforward.

#### People

Assignee:
Clare Saunders
Reporter:
John Parejko
Reviewers:
Krzysztof Findeisen
Watchers:
Clare Saunders, Eric Bellm, Jim Bosch, John Parejko, John Swinbank, Krzysztof Findeisen, Pierre Astier