Jim Bosch suggested I put a full proposal together. Based on his answers to my question above, I propose the following:
Here is the current schema for reference catalogs:
- coord: ICRS position of star on sky (an lsst.geom.SpherePoint)
- centroid: position of star on an exposure, if relevant (an lsst.afw.Point2D)
- hasCentroid: is centroid usable?
- referenceFilterName_flux: brightness in the specified reference catalog filter (Jy)
Note}}: the function lsst.afw.image.abMagFromFlux will convert flux in Jy to AB Magnitude.
- referenceFilterName_fluxSigma (optional): brightness standard deviation (Jy);
omitted if no data is available; possibly nan if data is available for some objects but not others
- camFlux: brightness in default camera filter (Jy); omitted if defaultFilter not specified
- camFluxSigma: brightness standard deviation for default camera filter;
omitted if defaultFilter not specified or standard deviation not available that filter
- cameraFilterName_camFlux: brightness in specified camera filter (Jy)
- cameraFilterName_camFluxSigma (optional): brightness standard deviation
in specified camera filter (Jy); omitted if no data is available;
possibly nan if data is available for some objects but not others
- photometric (optional): is the object usable for photometric calibration?
- resolved (optional): is the object spatially resolved?
- variable (optional): does the object have variable brightness?
Aside: I worry that camera fluxes no longer make sense, because we plan to process data for more than one camera at a time. I have no idea what to do about this so I'll not discuss it further.
Position error, proper motion and parallax
I propose these additional fields to support position error, proper motion and parallax. I believe the names are compatible with
RFC-333 and the units for parallax and proper motion match Gaia:
- coord_raErr, coord_decErr: position error (radians, to match coord)
- parallax: parallax (milliarcseconds)
- parallaxErr (milliarcseconds)
- pm_ra, pm_dec: proper motion (milliarcseconds/year, pmRa = dRA/dt * cos(dec))
- pm_raErr, pm_decErr: proper motion error (milliarcseconds/year)
Note that Gaia offers correlation between all of coord, parallax and proper motion. Of these correlations my guess is we will only be interested in the following:
- pm_radecErr (do we really want to keep track of this?)
All entries must have a known position and flux for at least one band, but other fields may be have unknown values. The question is how to handle this gracefully? Jim Bosch has argued for using "nan" for unknown values and Jim and John Parejko both suggest adding flag fields to make these easier to test for. I am willing to have flags for composite fields, but feel that flags add too much clutter for scalar fields and suggest users test "isfinite" for scalars. That results in something like the following:
Flags for composite fields:
- coordErrGood: coord_raErr, coord_decErr and (if present) coord_radecErr are finite
- pmGood: pm_ra and pm_dec are finite
- pmErrGood: pm_raErr, pm_decErr and (if present) pm_radecErr are finite
If we insist on flags for scalar fields, as well:
- parallaxGood: parallax is finite
- parallaxErrGood: parallax err is finite
We could use "good_" as a suffix if folks prefer. The underscore is important so we can use the standard case for the next word. This is crucial if we have flags for filters, as the filter name should have the same case in all fields.
Do we want to support reference catalogs that are missing some of the new fields? For now our code must continue to support catalogs that have none of these new fields, but what about new catalogs?
I think correlation fields, such as coord_radecErr, should be optional, because only Gaia provides these. CovarianceMatrixKey handles this gracefully, though it does not yet support the proposed error field names.
I think all data sources we are likely to use for future catalogs will have data for all the other new fields, so I propose we require them in all new catalogs.
I propose to file a ticket to add support to CovarianceMatrixKey (or a variant) that handles "Err" instead of "Sigma" and "ra/dec" axes. The support for "Err", at least, is required to implement