Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-20741

Generate Gaia DR2 reference catalog

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Story Points:
      2
    • Epic Link:
    • Sprint:
      AP F19-6 (November)
    • Team:
      Alert Production

      Description

      And make sure it is installed in the right place.

        Attachments

          Issue Links

            Activity

            Hide
            Parejkoj John Parejko added a comment -

            I've copied the gaia DR2 refcat over to lsst-dev and renamed a few things so that the symlink we made before now works. We should be able to use the refcat in processing as "gaia-dr2".

            I won't close this ticket until we've done a trial run with it, chosen a name for the directory, and actually moved the files into place.

            Show
            Parejkoj John Parejko added a comment - I've copied the gaia DR2 refcat over to lsst-dev and renamed a few things so that the symlink we made before now works. We should be able to use the refcat in processing as "gaia-dr2". I won't close this ticket until we've done a trial run with it, chosen a name for the directory, and actually moved the files into place.
            Hide
            Parejkoj John Parejko added a comment -

            Responding to Robert Lupton's comments on this Community post:

            My aim in generating this Gaia DR2 refcat was to produce something we could use to start exploring parallax and proper motion in the stack (in particular in jointcal). I didn't expect it to be used for photometry, and I note that we don't have Gaia colorterms available for any existing instruments. The final Gaia catalog should provide a much better photometric system. I'm surprised that you think it's worthwhile spending much time on this — would you mind explaining what your goal is?

            That said, I can work to provide flux errors if you need them. Following from my statements above, the issue here is really twofold:

            • The existing ingestion system is designed around fluxes and flux errors being generated based on magnitudes and magnitude errors in the external catalog. There's no easy way to avoid this, other than hard-coding a special case for Gaia. I can do that, but especially given @jbosch's suggested approach at restructuring our refcats on Community, I'm reluctant to do so unless it's really important.
            • Gaia publish their flux errors in terms of the instrumental flux, rather than the calibrated flux. They also publish a ratio of the instrumental flux error to the flux (phot_g_mean_flux_over_error). Would simply propagating that to the refcat be adequate for your purposes? It would require specialization of code that works with the output reference catalog to use that field, but would be a trivial step to generate, just requiring ~24 hours for my desktop to rebuild the catalog.

            I emailed the Gaia helpdesk this morning to ask them whether they plan to provide physical calibrated fluxes and errors and/or "symmetrized" magnitude errors for DR3.

            Show
            Parejkoj John Parejko added a comment - Responding to Robert Lupton 's comments on this Community post : My aim in generating this Gaia DR2 refcat was to produce something we could use to start exploring parallax and proper motion in the stack (in particular in jointcal). I didn't expect it to be used for photometry, and I note that we don't have Gaia colorterms available for any existing instruments. The final Gaia catalog should provide a much better photometric system. I'm surprised that you think it's worthwhile spending much time on this — would you mind explaining what your goal is? That said, I can work to provide flux errors if you need them. Following from my statements above, the issue here is really twofold: The existing ingestion system is designed around fluxes and flux errors being generated based on magnitudes and magnitude errors in the external catalog. There's no easy way to avoid this, other than hard-coding a special case for Gaia. I can do that, but especially given @jbosch's suggested approach at restructuring our refcats on Community , I'm reluctant to do so unless it's really important. Gaia publish their flux errors in terms of the instrumental flux, rather than the calibrated flux. They also publish a ratio of the instrumental flux error to the flux (phot_g_mean_flux_over_error). Would simply propagating that to the refcat be adequate for your purposes? It would require specialization of code that works with the output reference catalog to use that field, but would be a trivial step to generate, just requiring ~24 hours for my desktop to rebuild the catalog. I emailed the Gaia helpdesk this morning to ask them whether they plan to provide physical calibrated fluxes and errors and/or "symmetrized" magnitude errors for DR3.
            Hide
            rhl Robert Lupton added a comment -

            I found out about the lack of magnitude errors trying to use a S/N cut for astrometric solutions, so I think that adding the errors is certainly worthwhile.  Simply propagating errors from the flux errors (dmag = 2.5/ln(10) dflux/flux) is fine;  for any source with significant flux the errors are going to be close to symmetrical.  I think that we should use that formula and write the Gaia ref_cat in "standard form" (while awaiting the decision on Jim Bosch's proposal);  as this is a once-off conversion for any given input catalogue I don't see that having some special-case code is much of a problem.  Adding another column would move the onus onto pipeline code, and I would not be in favour.

            It would be good to include the BP and RP photometry at the same time (with errors) as e.g. it'll allow us to make refraction corrections to the astrometry even if we can't use the values for photometric calibrations.

            I don't think that it's likely that Gaia will add symmetrised errors;  at that point (S/N <~ 3) the pdf is not fully characterised by a simple variance (as the distribution is not Gaussian) and the gain is small.  As they point out, in that case you should be working in flux units, but we simply don't care.

            I think that the conversion from their flux units (e-/s) to calibrated fluxes and magnitudes (including AB, and thus Jy) is described in https://gea.esac.esa.int/archive/documentation/GDR2/Data_processing/chap_cu5pho/sec_cu5pho_calibr/ssec_cu5pho_calibr_extern.html (e.g. just above equation 5.36).

            Show
            rhl Robert Lupton added a comment - I found out about the lack of magnitude errors trying to use a S/N cut for astrometric solutions, so I think that adding the errors is certainly worthwhile.  Simply propagating errors from the flux errors ( dmag = 2.5/ln(10) dflux/flux ) is fine;  for any source with significant flux the errors are going to be close to symmetrical.  I think that we should use that formula and write the Gaia ref_cat in "standard form" (while awaiting the decision on Jim Bosch 's proposal);  as this is a once-off conversion for any given input catalogue I don't see that having some special-case code is much of a problem.  Adding another column would move the onus onto pipeline code, and I would not be in favour. It would be good to include the BP and RP photometry at the same time (with errors) as e.g. it'll allow us to make refraction corrections to the astrometry even if we can't use the values for photometric calibrations. I don't think that it's likely that Gaia will add symmetrised errors;  at that point (S/N <~ 3) the pdf is not fully characterised by a simple variance (as the distribution is not Gaussian) and the gain is small.  As they point out, in that case you should be working in flux units, but we simply don't care. I think that the conversion from their flux units (e-/s) to calibrated fluxes and magnitudes (including AB, and thus Jy) is described in https://gea.esac.esa.int/archive/documentation/GDR2/Data_processing/chap_cu5pho/sec_cu5pho_calibr/ssec_cu5pho_calibr_extern.html  (e.g. just above equation 5.36).
            Hide
            Parejkoj John Parejko added a comment -

            Closing this since the catalog now exists, and the "putting it in the right place" step has to happen post-RFC.

            Show
            Parejkoj John Parejko added a comment - Closing this since the catalog now exists, and the "putting it in the right place" step has to happen post-RFC.

              People

              • Assignee:
                Parejkoj John Parejko
                Reporter:
                swinbank John Swinbank
                Watchers:
                Hsin-Fang Chiang, John Parejko, John Swinbank, Robert Lupton
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel