Uploaded image for project: 'Request For Comments'
  1. Request For Comments
  2. RFC-271

Standardized schema for proper motions in reference catalogs

    Details

    • Type: RFC
    • Status: Implemented
    • Resolution: Done
    • Component/s: DM
    • Labels:
      None
    • Location:
      This RFC

      Description

      We will need to apply proper motion corrections (when available) to our reference catalogs when generating astrometric solutions. We have no current standardized schema for these columns. Following suggestions from Paul Price following the lead of Pan-STARRS, I propose the following column names and units.

      pmRa -- Proper motion in the RA coordinate (mas/yr; float32)
      pmRaErr -- Error in proper motion in the RA coordinate (mas/yr; float32)
      pmDec -- Proper motion in the Declination coordinate (mas/yr; float32)
      pmDecErr -- Error in proper motion in the Declination coordinate (mas/yr; float32)
      epoch -- Mean epoch (TAI MJD for which the position needs no proper motion correction; float64)
      

      Note that the definitions above allow direct application of the proper motion correction by multiplying pmRa and pmDec by the difference in epoch in units of years. If the angular distance travelled is to be calculated a correction term to pmRa of cos(Dec) is required (or one could do the dot product in Cartesian 3-space).

      If pmRaErr or pmDecErr are missing, zero error will be assumed. If any of pmRa, pmDec, or epoch are missing, no proper motion correction will be applied.

        Attachments

          Issue Links

            Activity

            krughoff Simon Krughoff created issue -
            krughoff Simon Krughoff made changes -
            Field Original Value New Value
            Link This issue is triggering DM-8837 [ DM-8837 ]
            Hide
            ktl Kian-Tat Lim added a comment -

            Is this parameterization likely to lead to problems (larger errors, for example) near the pole? That may not have been a problem for Pan-STARRS, but it might be one for us.

            Show
            ktl Kian-Tat Lim added a comment - Is this parameterization likely to lead to problems (larger errors, for example) near the pole? That may not have been a problem for Pan-STARRS, but it might be one for us.
            Hide
            krughoff Simon Krughoff added a comment -

            This parameterization is very common. Whether we store the angular velocity or the coordinate velocity, the PAL library expects proper motions in this parameterization (though in units of radians per year). I propose we use this and only propose a different parameterization if we find we are sensitive to numeric precision at the pole.

            Show
            krughoff Simon Krughoff added a comment - This parameterization is very common. Whether we store the angular velocity or the coordinate velocity, the PAL library expects proper motions in this parameterization (though in units of radians per year). I propose we use this and only propose a different parameterization if we find we are sensitive to numeric precision at the pole.
            price Paul Price made changes -
            Description We will need to apply proper motion corrections (when available) to our reference catalogs when generating astrometric solutions. We have no current standardized schema for these columns. Following suggestions from [~price] following the lead of Pan-STARRS, I propose the following column names and units.
            {noformat}
            pm_ra -- Proper motion in the RA coordinate (arcsec/yr)
            pm_ra_err -- Error in proper motion in the RA coordinate (arcsec/yr)
            pm_dec -- Proper motion in the Declination coordinate (arcsec/yr)
            pm_dec_err -- Error in proper motion in the Declination coordinate (arcsec/yr)
            epoch -- Mean epoch (seconds relative to the UNIX epoch)
            {noformat}

            Note that the definitions above allow direct application of the proper motion correction by multiplying {{pm_ra}} and {{pm_dec}} by the difference in epoch in units of years. If the angular distance travelled is to be calculated a correction term to {{pm_ra}} of {{cos(Dec)}} is required (or one could do the dot product in Cartesian 3-space).

            If {{pm_ra_err}} or {{pm_dec_err}} are missing, zero error will be assumed. If any of {{pm_ra}}, {{pm_dec}}, or {{epoch}} are missing, no proper motion correction will be applied.
            We will need to apply proper motion corrections (when available) to our reference catalogs when generating astrometric solutions. We have no current standardized schema for these columns. Following suggestions from [~price] following the lead of Pan-STARRS, I propose the following column names and units.
            {noformat}
            pm_ra -- Proper motion in the RA coordinate (arcsec/yr; float32)
            pm_ra_err -- Error in proper motion in the RA coordinate (arcsec/yr; float32)
            pm_dec -- Proper motion in the Declination coordinate (arcsec/yr; float32)
            pm_dec_err -- Error in proper motion in the Declination coordinate (arcsec/yr; float32)
            epoch -- Mean epoch (seconds relative to the UNIX epoch; int64)
            {noformat}

            Note that the definitions above allow direct application of the proper motion correction by multiplying {{pm_ra}} and {{pm_dec}} by the difference in epoch in units of years. If the angular distance travelled is to be calculated a correction term to {{pm_ra}} of {{cos(Dec)}} is required (or one could do the dot product in Cartesian 3-space).

            If {{pm_ra_err}} or {{pm_dec_err}} are missing, zero error will be assumed. If any of {{pm_ra}}, {{pm_dec}}, or {{epoch}} are missing, no proper motion correction will be applied.
            Hide
            tjenness Tim Jenness added a comment -

            "Mean epoch" seems a bit open to interpretation, do you mean "the epoch for which the coordinates specified in the ra and dec columns are correct"? The units of unix epoch seconds seem a bit weird to me for an astronomical epoch but I'm sure you'll say it's more efficient to apply the correction that way.

            Show
            tjenness Tim Jenness added a comment - "Mean epoch" seems a bit open to interpretation, do you mean "the epoch for which the coordinates specified in the ra and dec columns are correct"? The units of unix epoch seconds seem a bit weird to me for an astronomical epoch but I'm sure you'll say it's more efficient to apply the correction that way.
            Hide
            krughoff Simon Krughoff added a comment -

            do you mean "the epoch for which the coordinates specified in the ra and dec columns are correct"?

            I took the wording verbatim from a conversation with Paul Price, so he may want to comment, but that was my understanding.

            As for the units of the epoch, I like units of seconds since it removes most of the ambiguity of what time system we are in. The zeropoint is up for debate. Again, this was motivated by the Pan-STARRS choice.

            Show
            krughoff Simon Krughoff added a comment - do you mean "the epoch for which the coordinates specified in the ra and dec columns are correct"? I took the wording verbatim from a conversation with Paul Price , so he may want to comment, but that was my understanding. As for the units of the epoch, I like units of seconds since it removes most of the ambiguity of what time system we are in. The zeropoint is up for debate. Again, this was motivated by the Pan-STARRS choice.
            Hide
            price Paul Price added a comment -

            My understanding is the same as Simon's.

            Show
            price Paul Price added a comment - My understanding is the same as Simon's.
            Hide
            tjenness Tim Jenness added a comment -

            Simon Krughoff are you adopting this RFC? I would prefer that when it comes time to document the columns that we are a bit clearer about definitions.

            Show
            tjenness Tim Jenness added a comment - Simon Krughoff are you adopting this RFC? I would prefer that when it comes time to document the columns that we are a bit clearer about definitions.
            Hide
            krughoff Simon Krughoff added a comment -

            Tim Jenness which definitions need clarifying for you? I'd rather we be clear on what I'm saying when we adopt this so they are not open to reinterpretation later.

            Show
            krughoff Simon Krughoff added a comment - Tim Jenness which definitions need clarifying for you? I'd rather we be clear on what I'm saying when we adopt this so they are not open to reinterpretation later.
            Hide
            tjenness Tim Jenness added a comment -

            epoch. It didn't seem like a clear definition to me (as I mentioned above).

            Show
            tjenness Tim Jenness added a comment - epoch . It didn't seem like a clear definition to me (as I mentioned above).
            Hide
            tjenness Tim Jenness added a comment -

            The DPDD object table uses mas/yr so should we consider matching that? The epoch definition is "Time at which the object was at position psRadec." which seems very clear to me. Object uses TAI MJD Double for epoch.

            Show
            tjenness Tim Jenness added a comment - The DPDD object table uses mas/yr so should we consider matching that? The epoch definition is "Time at which the object was at position psRadec." which seems very clear to me. Object uses TAI MJD Double for epoch.
            krughoff Simon Krughoff made changes -
            Description We will need to apply proper motion corrections (when available) to our reference catalogs when generating astrometric solutions. We have no current standardized schema for these columns. Following suggestions from [~price] following the lead of Pan-STARRS, I propose the following column names and units.
            {noformat}
            pm_ra -- Proper motion in the RA coordinate (arcsec/yr; float32)
            pm_ra_err -- Error in proper motion in the RA coordinate (arcsec/yr; float32)
            pm_dec -- Proper motion in the Declination coordinate (arcsec/yr; float32)
            pm_dec_err -- Error in proper motion in the Declination coordinate (arcsec/yr; float32)
            epoch -- Mean epoch (seconds relative to the UNIX epoch; int64)
            {noformat}

            Note that the definitions above allow direct application of the proper motion correction by multiplying {{pm_ra}} and {{pm_dec}} by the difference in epoch in units of years. If the angular distance travelled is to be calculated a correction term to {{pm_ra}} of {{cos(Dec)}} is required (or one could do the dot product in Cartesian 3-space).

            If {{pm_ra_err}} or {{pm_dec_err}} are missing, zero error will be assumed. If any of {{pm_ra}}, {{pm_dec}}, or {{epoch}} are missing, no proper motion correction will be applied.
            We will need to apply proper motion corrections (when available) to our reference catalogs when generating astrometric solutions. We have no current standardized schema for these columns. Following suggestions from [~price] following the lead of Pan-STARRS, I propose the following column names and units.
            {noformat}
            pm_ra -- Proper motion in the RA coordinate (mas/yr; float32)
            pm_ra_err -- Error in proper motion in the RA coordinate (mas/yr; float32)
            pm_dec -- Proper motion in the Declination coordinate (mas/yr; float32)
            pm_dec_err -- Error in proper motion in the Declination coordinate (mas/yr; float32)
            epoch -- Mean epoch (MJD (TAI) for which the position needs no proper motion correction; float32)
            {noformat}

            Note that the definitions above allow direct application of the proper motion correction by multiplying {{pm_ra}} and {{pm_dec}} by the difference in epoch in units of years. If the angular distance travelled is to be calculated a correction term to {{pm_ra}} of {{cos(Dec)}} is required (or one could do the dot product in Cartesian 3-space).

            If {{pm_ra_err}} or {{pm_dec_err}} are missing, zero error will be assumed. If any of {{pm_ra}}, {{pm_dec}}, or {{epoch}} are missing, no proper motion correction will be applied.
            Hide
            krughoff Simon Krughoff added a comment -

            I have edited the description to reflect the DPDD descriptions.

            Show
            krughoff Simon Krughoff added a comment - I have edited the description to reflect the DPDD descriptions.
            Hide
            tjenness Tim Jenness added a comment -

            TAI MJD should be a double, not float.

            Show
            tjenness Tim Jenness added a comment - TAI MJD should be a double, not float.
            krughoff Simon Krughoff made changes -
            Description We will need to apply proper motion corrections (when available) to our reference catalogs when generating astrometric solutions. We have no current standardized schema for these columns. Following suggestions from [~price] following the lead of Pan-STARRS, I propose the following column names and units.
            {noformat}
            pm_ra -- Proper motion in the RA coordinate (mas/yr; float32)
            pm_ra_err -- Error in proper motion in the RA coordinate (mas/yr; float32)
            pm_dec -- Proper motion in the Declination coordinate (mas/yr; float32)
            pm_dec_err -- Error in proper motion in the Declination coordinate (mas/yr; float32)
            epoch -- Mean epoch (MJD (TAI) for which the position needs no proper motion correction; float32)
            {noformat}

            Note that the definitions above allow direct application of the proper motion correction by multiplying {{pm_ra}} and {{pm_dec}} by the difference in epoch in units of years. If the angular distance travelled is to be calculated a correction term to {{pm_ra}} of {{cos(Dec)}} is required (or one could do the dot product in Cartesian 3-space).

            If {{pm_ra_err}} or {{pm_dec_err}} are missing, zero error will be assumed. If any of {{pm_ra}}, {{pm_dec}}, or {{epoch}} are missing, no proper motion correction will be applied.
            We will need to apply proper motion corrections (when available) to our reference catalogs when generating astrometric solutions. We have no current standardized schema for these columns. Following suggestions from [~price] following the lead of Pan-STARRS, I propose the following column names and units.
            {noformat}
            pm_ra -- Proper motion in the RA coordinate (mas/yr; float32)
            pm_ra_err -- Error in proper motion in the RA coordinate (mas/yr; float32)
            pm_dec -- Proper motion in the Declination coordinate (mas/yr; float32)
            pm_dec_err -- Error in proper motion in the Declination coordinate (mas/yr; float32)
            epoch -- Mean epoch (MJD (TAI) for which the position needs no proper motion correction; float64)
            {noformat}

            Note that the definitions above allow direct application of the proper motion correction by multiplying {{pm_ra}} and {{pm_dec}} by the difference in epoch in units of years. If the angular distance travelled is to be calculated a correction term to {{pm_ra}} of {{cos(Dec)}} is required (or one could do the dot product in Cartesian 3-space).

            If {{pm_ra_err}} or {{pm_dec_err}} are missing, zero error will be assumed. If any of {{pm_ra}}, {{pm_dec}}, or {{epoch}} are missing, no proper motion correction will be applied.
            krughoff Simon Krughoff made changes -
            Description We will need to apply proper motion corrections (when available) to our reference catalogs when generating astrometric solutions. We have no current standardized schema for these columns. Following suggestions from [~price] following the lead of Pan-STARRS, I propose the following column names and units.
            {noformat}
            pm_ra -- Proper motion in the RA coordinate (mas/yr; float32)
            pm_ra_err -- Error in proper motion in the RA coordinate (mas/yr; float32)
            pm_dec -- Proper motion in the Declination coordinate (mas/yr; float32)
            pm_dec_err -- Error in proper motion in the Declination coordinate (mas/yr; float32)
            epoch -- Mean epoch (MJD (TAI) for which the position needs no proper motion correction; float64)
            {noformat}

            Note that the definitions above allow direct application of the proper motion correction by multiplying {{pm_ra}} and {{pm_dec}} by the difference in epoch in units of years. If the angular distance travelled is to be calculated a correction term to {{pm_ra}} of {{cos(Dec)}} is required (or one could do the dot product in Cartesian 3-space).

            If {{pm_ra_err}} or {{pm_dec_err}} are missing, zero error will be assumed. If any of {{pm_ra}}, {{pm_dec}}, or {{epoch}} are missing, no proper motion correction will be applied.
            We will need to apply proper motion corrections (when available) to our reference catalogs when generating astrometric solutions. We have no current standardized schema for these columns. Following suggestions from [~price] following the lead of Pan-STARRS, I propose the following column names and units.
            {noformat}
            pm_ra -- Proper motion in the RA coordinate (mas/yr; float32)
            pm_ra_err -- Error in proper motion in the RA coordinate (mas/yr; float32)
            pm_dec -- Proper motion in the Declination coordinate (mas/yr; float32)
            pm_dec_err -- Error in proper motion in the Declination coordinate (mas/yr; float32)
            epoch -- Mean epoch (TAI MJD for which the position needs no proper motion correction; float64)
            {noformat}

            Note that the definitions above allow direct application of the proper motion correction by multiplying {{pm_ra}} and {{pm_dec}} by the difference in epoch in units of years. If the angular distance travelled is to be calculated a correction term to {{pm_ra}} of {{cos(Dec)}} is required (or one could do the dot product in Cartesian 3-space).

            If {{pm_ra_err}} or {{pm_dec_err}} are missing, zero error will be assumed. If any of {{pm_ra}}, {{pm_dec}}, or {{epoch}} are missing, no proper motion correction will be applied.
            Hide
            krughoff Simon Krughoff added a comment -

            Tim Jenness are you happy with this now?

            Show
            krughoff Simon Krughoff added a comment - Tim Jenness are you happy with this now?
            Hide
            tjenness Tim Jenness added a comment -

            Yes.

            Show
            tjenness Tim Jenness added a comment - Yes.
            Hide
            tjenness Tim Jenness added a comment -

            Simon Krughoff are you going to adopt this RFC then?

            Show
            tjenness Tim Jenness added a comment - Simon Krughoff are you going to adopt this RFC then?
            Hide
            krughoff Simon Krughoff added a comment -

            Thanks for the prodding. I just wanted to give Paul Price a chance to react and he did so on slack. There is no dissent.

            Show
            krughoff Simon Krughoff added a comment - Thanks for the prodding. I just wanted to give Paul Price a chance to react and he did so on slack. There is no dissent.
            krughoff Simon Krughoff made changes -
            Resolution Done [ 10000 ]
            Status Proposed [ 10805 ] Adopted [ 10806 ]
            Parejkoj John Parejko made changes -
            Link This issue relates to DM-8828 [ DM-8828 ]
            krughoff Simon Krughoff made changes -
            Description We will need to apply proper motion corrections (when available) to our reference catalogs when generating astrometric solutions. We have no current standardized schema for these columns. Following suggestions from [~price] following the lead of Pan-STARRS, I propose the following column names and units.
            {noformat}
            pm_ra -- Proper motion in the RA coordinate (mas/yr; float32)
            pm_ra_err -- Error in proper motion in the RA coordinate (mas/yr; float32)
            pm_dec -- Proper motion in the Declination coordinate (mas/yr; float32)
            pm_dec_err -- Error in proper motion in the Declination coordinate (mas/yr; float32)
            epoch -- Mean epoch (TAI MJD for which the position needs no proper motion correction; float64)
            {noformat}

            Note that the definitions above allow direct application of the proper motion correction by multiplying {{pm_ra}} and {{pm_dec}} by the difference in epoch in units of years. If the angular distance travelled is to be calculated a correction term to {{pm_ra}} of {{cos(Dec)}} is required (or one could do the dot product in Cartesian 3-space).

            If {{pm_ra_err}} or {{pm_dec_err}} are missing, zero error will be assumed. If any of {{pm_ra}}, {{pm_dec}}, or {{epoch}} are missing, no proper motion correction will be applied.
            We will need to apply proper motion corrections (when available) to our reference catalogs when generating astrometric solutions. We have no current standardized schema for these columns. Following suggestions from [~price] following the lead of Pan-STARRS, I propose the following column names and units.
            {noformat}
            pmRa -- Proper motion in the RA coordinate (mas/yr; float32)
            pmRaErr -- Error in proper motion in the RA coordinate (mas/yr; float32)
            pmDec -- Proper motion in the Declination coordinate (mas/yr; float32)
            pmDecErr -- Error in proper motion in the Declination coordinate (mas/yr; float32)
            epoch -- Mean epoch (TAI MJD for which the position needs no proper motion correction; float64)
            {noformat}

            Note that the definitions above allow direct application of the proper motion correction by multiplying {{pmRa}} and {{pmDec}} by the difference in epoch in units of years. If the angular distance travelled is to be calculated a correction term to {{pmRa}} of {{cos(Dec)}} is required (or one could do the dot product in Cartesian 3-space).

            If {{pmRaErr}} or {{pmDecErr}} are missing, zero error will be assumed. If any of {{pmRa}}, {{pmDec}}, or {{epoch}} are missing, no proper motion correction will be applied.
            Hide
            jbosch Jim Bosch added a comment -

            Sorry I didn't notice this further, but while I have no objection to the parameterization proposed here, we should use use names like "pm_ra", "pm_dec", "pm_raErr", and "pm_decErr", for both consistency with how other aggregates are named in our schemas and for compatibility with FunctorKey classes (which let us enforce at least some of the naming conventions at a code level).

            Using CovarianceMatrixKey to abstract the naming for the uncertainty fields would result in "pm_raSigma" and "pm_decSigma" until RFC-333 is implemented, but I think that's okay, as internal consistency is probably as important as future-proofness.  That said, going with "Err" rather than "Sigma" now for reference catalogs may be better, even if it makes it impossible to use CovarianceMatrixKey right now, because updating all of our extant reference catalogs would be an unpleasant addition to the implementation of RFC-333.

            Show
            jbosch Jim Bosch added a comment - Sorry I didn't notice this further, but while I have no objection to the parameterization proposed here, we should use use names like "pm_ra", "pm_dec", "pm_raErr", and "pm_decErr", for both consistency with how other aggregates are named in our schemas and for compatibility with FunctorKey classes (which let us enforce at least some of the naming conventions at a code level). Using CovarianceMatrixKey to abstract the naming for the uncertainty fields would result in "pm_raSigma" and "pm_decSigma" until RFC-333 is implemented, but I think that's okay, as internal consistency is probably as important as future-proofness.  That said, going with "Err" rather than "Sigma" now for reference catalogs may be better, even if it makes it impossible to use CovarianceMatrixKey right now, because updating all of our extant reference catalogs would be an unpleasant addition to the implementation of RFC-333 .
            jbosch Jim Bosch made changes -
            Resolution Done [ 10000 ]
            Status Adopted [ 10806 ] Flagged [ 10606 ]
            jbosch Jim Bosch made changes -
            Risk Score 0
            jbosch Jim Bosch made changes -
            Status Flagged [ 10606 ] Proposed [ 10805 ]
            Hide
            rowen Russell Owen added a comment -

            +1 to Jim's recent suggested change. I am in favor of consistency with RFC-333. However, I feel very strongly that whatever name we pick now we should stick with, which apparently means not using CovarianceMatrixKey until a version appears that can handle the "Err" suffix. It will be a lot less work than changing our catalogs later.

            Note that we also need names for RA and Dec uncertainty of the "coord" field. I assume these will be coord_raErr and coord_decErr.

            I plan to to add a field for parallax now, even if we don't implement the correction yet. Will we want uncertainty in parallax as well?

            Show
            rowen Russell Owen added a comment - +1 to Jim's recent suggested change. I am in favor of consistency with RFC-333 . However, I feel very strongly that whatever name we pick now we should stick with, which apparently means not using CovarianceMatrixKey until a version appears that can handle the "Err" suffix. It will be a lot less work than changing our catalogs later. Note that we also need names for RA and Dec uncertainty of the "coord" field. I assume these will be coord_raErr and coord_decErr . I plan to to add a field for parallax now, even if we don't implement the correction yet. Will we want uncertainty in parallax as well?
            Hide
            jbosch Jim Bosch added a comment -

            Note that we also need names for RA and Dec uncertainty of the "coord" field. I assume these will be coord_raErr and coord_decErr.

            I plan to to add a field for parallax now, even if we don't implement the correction yet. Will we want uncertainty in parallax as well?

            Probably, unless the way to express that uncertainty is somehow controversial (though if it is, I'd probably just pick whatever Gaia does).

            Show
            jbosch Jim Bosch added a comment - Note that we also need names for RA and Dec uncertainty of the "coord" field. I assume these will be  coord_raErr  and  coord_decErr . I plan to to add a field for parallax now, even if we don't implement the correction yet. Will we want uncertainty in parallax as well? Probably, unless the way to express that uncertainty is somehow controversial (though if it is, I'd probably just pick whatever Gaia does).
            Hide
            rowen Russell Owen added a comment -

            Another question, since this has been re-opened: do we really want to use 0 for error columns if the column in question is not available in the input catalog? That is what the initial proposal suggests for proper motion errors. We also have to deal with position errors and parallax errors.

            Three obvious options if an error field is unavailable in the input catalog:

            • Set the field to 0, as per the initial proposal.
            • Set the field to nan.
            • Do not provide the field at all.
            • Flag the field as having invalid values, e.g. in the table metadata.

            I lean towards using "nan" for unknown values because:

            • Some input catalogs may have a subset of objects for which errors are unknown. In that case "nan" seems the only reasonable value for the unknown errors, since the other objects will have valid values for the errors.
            • Since we may have "nan" for errors for some objects, we may as well be consistent and always use "nan" for "unknown error".

            I'm personally against omitting or flagging the fields, as I think it makes code too complicated. But I admit that writing code in such a way that it can gracefully handle "nan" is also a hassle.

            Show
            rowen Russell Owen added a comment - Another question, since this has been re-opened: do we really want to use 0 for error columns if the column in question is not available in the input catalog? That is what the initial proposal suggests for proper motion errors. We also have to deal with position errors and parallax errors. Three obvious options if an error field is unavailable in the input catalog: Set the field to 0, as per the initial proposal. Set the field to nan. Do not provide the field at all. Flag the field as having invalid values, e.g. in the table metadata. I lean towards using "nan" for unknown values because: Some input catalogs may have a subset of objects for which errors are unknown. In that case "nan" seems the only reasonable value for the unknown errors, since the other objects will have valid values for the errors. Since we may have "nan" for errors for some objects, we may as well be consistent and always use "nan" for "unknown error". I'm personally against omitting or flagging the fields, as I think it makes code too complicated. But I admit that writing code in such a way that it can gracefully handle "nan" is also a hassle.
            Hide
            jbosch Jim Bosch added a comment -

            I don't think zero for missing values works; that seems quite backwards.

            I'd prefer not having the fields at all when a value is never provided for a particular catalog (but only weakly; I'm open to being persuaded that this makes the implementation or calling code more complex).

            I'd prefer using both NaN and flags when a field is present for some objects, but not all.  I find testing flags a bit cleaner than testing NaNs, especially when a single flag can be used to test for the presence or absence of multiple fields (e.g. "are there proper motions at all").

            Show
            jbosch Jim Bosch added a comment - I don't think zero for missing values works; that seems quite backwards. I'd prefer not having the fields at all when a value is never provided for a particular catalog (but only weakly; I'm open to being persuaded that this makes the implementation or calling code more complex). I'd prefer using both NaN and flags when a field is present for some objects, but not all.  I find testing flags a bit cleaner than testing NaNs, especially when a single flag can be used to test for the presence or absence of multiple fields (e.g. "are there proper motions at all").
            Hide
            rowen Russell Owen added a comment -

            Note that RFC-368 proposes to always include the error fields and to suggests using 0 for unknown values. But John Parejko feels that whatever we decide here for unknown values should apply to that RFC as well.

            Show
            rowen Russell Owen added a comment - Note that RFC-368 proposes to always include the error fields and to suggests using 0 for unknown values. But John Parejko feels that whatever we decide here for unknown values should apply to that RFC as well.
            Hide
            Parejkoj John Parejko added a comment -

            I'd suggest we have something like goodAstrometry_flag, goodPhotometry_flag, goodProperMotion_flag, and goodParallax_flag (or the inverse e.g. "badX_flag" if we prefer), so that users can easily select objects that are useful for that particular thing (respectively non-NaN coordErr, non-Nan flux and fluxErr, non-NaN for all the pm fields). Otherwise, it's hard for a catalog user to know what fields to check for NaN-ness: that knowledge should be incorporated by the author of the catalog.

            My response here is related to from DM-7099 and this Community discussion: https://community.lsst.org/t/additional-summary-flags-during-single-frame-and-centroid-processing/901

            Show
            Parejkoj John Parejko added a comment - I'd suggest we have something like goodAstrometry_flag , goodPhotometry_flag , goodProperMotion_flag , and goodParallax_flag (or the inverse e.g. "badX_flag" if we prefer), so that users can easily select objects that are useful for that particular thing (respectively non-NaN coordErr , non-Nan flux and fluxErr , non-NaN for all the pm fields). Otherwise, it's hard for a catalog user to know what fields to check for NaN-ness: that knowledge should be incorporated by the author of the catalog. My response here is related to from DM-7099 and this Community discussion: https://community.lsst.org/t/additional-summary-flags-during-single-frame-and-centroid-processing/901
            Hide
            rowen Russell Owen added a comment - - edited

            Jim Bosch and John Parejko have convinced me that adding flags is a good idea. I agree with Jim Bosch that 0 is a bad idea for unknown values.

            Please note that the existing reference catalog flag fields have no _flag suffix: "hasCentroid", "photometric", "resolved", "variable", so I would be inclined not to use that suffix for new flag fields.

            Show
            rowen Russell Owen added a comment - - edited Jim Bosch and John Parejko have convinced me that adding flags is a good idea. I agree with Jim Bosch that 0 is a bad idea for unknown values. Please note that the existing reference catalog flag fields have no _flag suffix: "hasCentroid", "photometric", "resolved", "variable", so I would be inclined not to use that suffix for new flag fields.
            Hide
            rowen Russell Owen added a comment - - edited

            A few questions about flags for the catalogs we are likely to use:

            • If a valid value is given for a measured quantity (e.g. position, a flux in a band, proper motion or parallax), is a valid error also always available? I hope so, because if so then we should always provide error fields (if the quantity field is present) and a single flag will suffice for both the measured quantity and the associated error. Otherwise we will presumably need one flag for the value and another for the error.
            • If a measured quantity is available, are valid values available for all entries in the catalog? If so, we might not need flags. This seems less likely, but I'm not familiar enough with the catalogs to know.

            Note that flux is provided for different bands, so if we want a flag for photometry we'll probably need one flag for each band.

            Another question: what about covariance (correlation) for position and/or proper motion? Gaia provides a number of these, including RA/Dec and mixtures of that with proper motion. SDSS does not

            Show
            rowen Russell Owen added a comment - - edited A few questions about flags for the catalogs we are likely to use: If a valid value is given for a measured quantity (e.g. position, a flux in a band, proper motion or parallax), is a valid error also always available? I hope so, because if so then we should always provide error fields (if the quantity field is present) and a single flag will suffice for both the measured quantity and the associated error. Otherwise we will presumably need one flag for the value and another for the error. If a measured quantity is available, are valid values available for all entries in the catalog? If so, we might not need flags. This seems less likely, but I'm not familiar enough with the catalogs to know. Note that flux is provided for different bands, so if we want a flag for photometry we'll probably need one flag for each band. Another question: what about covariance (correlation) for position and/or proper motion? Gaia provides a number of these, including RA/Dec and mixtures of that with proper motion. SDSS does not
            rowen Russell Owen made changes -
            Comment [ Another question: what about covariance (correlation) for position and/or proper motion? Gaia provides a number of these, including RA/Dec and mixtures of that with proper motion. SDSS does not. ]
            Hide
            jbosch Jim Bosch added a comment -

            If a valid value is given for a measured quantity (e.g. position, a flux in a band, proper motion or parallax), is a valid error also always available?

            Sadly, I don't think we can realistically expect this to always be true.

            If a measured quantity is available, are valid values available for all entries in the catalog? If so, we might not need flags. This seems less likely, but I'm not familiar enough with the catalogs to know.

            I don't think we can make this assumption either.

            Another question: what about covariance (correlation) for position and/or proper motion? Gaia provides a number of these, including RA/Dec and mixtures of that with proper motion. SDSS does not.

            I think it'd be best to include these when available.  Note that CovarianceMatrixKey can probably make it easier to deal with the fact that some catalogs will have off-diagonal covariance terms while others will not.

            Show
            jbosch Jim Bosch added a comment - If a valid value is given for a measured quantity (e.g. position, a flux in a band, proper motion or parallax), is a valid error also always available? Sadly, I don't think we can realistically expect this to always be true. If a measured quantity is available, are valid values available for all entries in the catalog? If so, we might not need flags. This seems less likely, but I'm not familiar enough with the catalogs to know. I don't think we can make this assumption either. Another question: what about covariance (correlation) for position and/or proper motion? Gaia provides a number of these, including RA/Dec and mixtures of that with proper motion. SDSS does not. I think it'd be best to include these when available.  Note that CovarianceMatrixKey can probably make it easier to deal with the fact that some catalogs will have off-diagonal covariance terms while others will not.
            Hide
            rowen Russell Owen added a comment -

            Jim Bosch suggested I put a full proposal together. Based on his answers to my question above, I propose the following:

            Current Schema

            Here is the current schema for reference catalogs:

            • coord: ICRS position of star on sky (an lsst.geom.SpherePoint)
            • centroid: position of star on an exposure, if relevant (an lsst.afw.Point2D)
            • hasCentroid: is centroid usable?
            • referenceFilterName_flux: brightness in the specified reference catalog filter (Jy)
              Note}}: the function lsst.afw.image.abMagFromFlux will convert flux in Jy to AB Magnitude.
            • referenceFilterName_fluxSigma (optional): brightness standard deviation (Jy);
              omitted if no data is available; possibly nan if data is available for some objects but not others
            • camFlux: brightness in default camera filter (Jy); omitted if defaultFilter not specified
            • camFluxSigma: brightness standard deviation for default camera filter;
              omitted if defaultFilter not specified or standard deviation not available that filter
            • cameraFilterName_camFlux: brightness in specified camera filter (Jy)
            • cameraFilterName_camFluxSigma (optional): brightness standard deviation
              in specified camera filter (Jy); omitted if no data is available;
              possibly nan if data is available for some objects but not others
            • photometric (optional): is the object usable for photometric calibration?
            • resolved (optional): is the object spatially resolved?
            • variable (optional): does the object have variable brightness?

            Aside: I worry that camera fluxes no longer make sense, because we plan to process data for more than one camera at a time. I have no idea what to do about this so I'll not discuss it further.

            Position error, proper motion and parallax

            I propose these additional fields to support position error, proper motion and parallax. I believe the names are compatible with RFC-333 and the units for parallax and proper motion match Gaia:

            • coord_raErr, coord_decErr: position error (radians, to match coord)
            • parallax: parallax (milliarcseconds)
            • parallaxErr (milliarcseconds)
            • pm_ra, pm_dec: proper motion (milliarcseconds/year, pmRa = dRA/dt * cos(dec))
            • pm_raErr, pm_decErr: proper motion error (milliarcseconds/year)
              Note that Gaia offers correlation between all of coord, parallax and proper motion. Of these correlations my guess is we will only be interested in the following:
            • coord_radecErr
            • pm_radecErr (do we really want to keep track of this?)

            Unknown values

            All entries must have a known position and flux for at least one band, but other fields may be have unknown values. The question is how to handle this gracefully? Jim Bosch has argued for using "nan" for unknown values and Jim and John Parejko both suggest adding flag fields to make these easier to test for. I am willing to have flags for composite fields, but feel that flags add too much clutter for scalar fields and suggest users test "isfinite" for scalars. That results in something like the following:

            Flags for composite fields:

            • coordErrGood: coord_raErr, coord_decErr and (if present) coord_radecErr are finite
            • pmGood: pm_ra and pm_dec are finite
            • pmErrGood: pm_raErr, pm_decErr and (if present) pm_radecErr are finite

            If we insist on flags for scalar fields, as well:

            • referenceFilterName_fluxGood
            • referenceFilterName_fluxErrGood
            • camFluxGood
            • camFluxErrGood
            • cameraFilterName_camFluxGood
            • cameraFilterName_camFluxErrGood
            • parallaxGood: parallax is finite
            • parallaxErrGood: parallax err is finite

            We could use "good_" as a suffix if folks prefer. The underscore is important so we can use the standard case for the next word. This is crucial if we have flags for filters, as the filter name should have the same case in all fields.

            Missing fields

            Do we want to support reference catalogs that are missing some of the new fields? For now our code must continue to support catalogs that have none of these new fields, but what about new catalogs?

            I think correlation fields, such as coord_radecErr, should be optional, because only Gaia provides these. CovarianceMatrixKey handles this gracefully, though it does not yet support the proposed error field names.

            I think all data sources we are likely to use for future catalogs will have data for all the other new fields, so I propose we require them in all new catalogs.

            Other work

            I propose to file a ticket to add support to CovarianceMatrixKey (or a variant) that handles "Err" instead of "Sigma" and "ra/dec" axes. The support for "Err", at least, is required to implement RFC-133.

            Show
            rowen Russell Owen added a comment - Jim Bosch suggested I put a full proposal together. Based on his answers to my question above, I propose the following: Current Schema Here is the current schema for reference catalogs: coord : ICRS position of star on sky (an lsst.geom.SpherePoint) centroid : position of star on an exposure, if relevant (an lsst.afw.Point2D) hasCentroid : is centroid usable? referenceFilterName _flux : brightness in the specified reference catalog filter (Jy) Note}}: the function lsst.afw.image.abMagFromFlux will convert flux in Jy to AB Magnitude. referenceFilterName _fluxSigma (optional): brightness standard deviation (Jy); omitted if no data is available; possibly nan if data is available for some objects but not others camFlux : brightness in default camera filter (Jy); omitted if defaultFilter not specified camFluxSigma : brightness standard deviation for default camera filter; omitted if defaultFilter not specified or standard deviation not available that filter cameraFilterName _camFlux : brightness in specified camera filter (Jy) cameraFilterName _camFluxSigma (optional): brightness standard deviation in specified camera filter (Jy); omitted if no data is available; possibly nan if data is available for some objects but not others photometric (optional): is the object usable for photometric calibration? resolved (optional): is the object spatially resolved? variable (optional): does the object have variable brightness? Aside: I worry that camera fluxes no longer make sense, because we plan to process data for more than one camera at a time. I have no idea what to do about this so I'll not discuss it further. Position error, proper motion and parallax I propose these additional fields to support position error, proper motion and parallax. I believe the names are compatible with RFC-333 and the units for parallax and proper motion match Gaia: coord_raErr, coord_decErr : position error (radians, to match coord) parallax : parallax (milliarcseconds) parallaxErr (milliarcseconds) pm_ra, pm_dec : proper motion (milliarcseconds/year, pmRa = dRA/dt * cos(dec)) pm_raErr, pm_decErr : proper motion error (milliarcseconds/year) Note that Gaia offers correlation between all of coord, parallax and proper motion. Of these correlations my guess is we will only be interested in the following: coord_radecErr pm_radecErr (do we really want to keep track of this?) Unknown values All entries must have a known position and flux for at least one band, but other fields may be have unknown values. The question is how to handle this gracefully? Jim Bosch has argued for using "nan" for unknown values and Jim and John Parejko both suggest adding flag fields to make these easier to test for. I am willing to have flags for composite fields, but feel that flags add too much clutter for scalar fields and suggest users test "isfinite" for scalars. That results in something like the following: Flags for composite fields: coordErrGood : coord_raErr, coord_decErr and (if present) coord_radecErr are finite pmGood : pm_ra and pm_dec are finite pmErrGood : pm_raErr, pm_decErr and (if present) pm_radecErr are finite If we insist on flags for scalar fields, as well: referenceFilterName _fluxGood referenceFilterName _fluxErrGood camFluxGood camFluxErrGood cameraFilterName _camFluxGood cameraFilterName _camFluxErrGood parallaxGood : parallax is finite parallaxErrGood : parallax err is finite We could use "good_" as a suffix if folks prefer. The underscore is important so we can use the standard case for the next word. This is crucial if we have flags for filters, as the filter name should have the same case in all fields. Missing fields Do we want to support reference catalogs that are missing some of the new fields? For now our code must continue to support catalogs that have none of these new fields, but what about new catalogs? I think correlation fields, such as coord_radecErr , should be optional, because only Gaia provides these. CovarianceMatrixKey handles this gracefully, though it does not yet support the proposed error field names. I think all data sources we are likely to use for future catalogs will have data for all the other new fields, so I propose we require them in all new catalogs. Other work I propose to file a ticket to add support to CovarianceMatrixKey (or a variant) that handles "Err" instead of "Sigma" and "ra/dec" axes. The support for "Err", at least, is required to implement RFC-133 .
            rowen Russell Owen made changes -
            Watchers Jim Bosch, John Parejko, Kian-Tat Lim, Paul Price, Russell Owen, Simon Krughoff, Tim Jenness [ Jim Bosch, John Parejko, Kian-Tat Lim, Paul Price, Russell Owen, Simon Krughoff, Tim Jenness ] Jim Bosch, John Parejko, Kian-Tat Lim, Paul Price, Robert Lupton, Russell Owen, Simon Krughoff, Tim Jenness [ Jim Bosch, John Parejko, Kian-Tat Lim, Paul Price, Robert Lupton, Russell Owen, Simon Krughoff, Tim Jenness ]
            Hide
            rowen Russell Owen added a comment -
            Show
            rowen Russell Owen added a comment - Also, for reference, here is Gaia's data model:  http://gea.esac.esa.int/archive/documentation/GDR1/datamodel/Ch1/gaia_source.html
            Hide
            tjenness Tim Jenness added a comment -

            Since this RFC got reopened, can someone please set the planned end date to a new value?

            Show
            tjenness Tim Jenness added a comment - Since this RFC got reopened, can someone please set the planned end date to a new value?
            Hide
            jbosch Jim Bosch added a comment -

            Russell Owen, could you remind me why we have:

            • centroid: position of star on an exposure, if relevant (an lsst.afw.Point2D)
            • hasCentroid: is centroid usable?

            Are these just workspace created for algorithms that use the reference catalog?  If so, are they actually included in the on-disk representation?

            I have similar concerns about the various camFlux columns; mixing quantities derived from the image we're processing with those from the reference catalog seems like bad design, though I suspect it's a workaround for joins and schema manipulation of afw.tables being clunky.

             

            More generally, I started to try to put together a counter-proposal with mostly minor changes, and I found myself quite dissatisfied with our options for dealing with units in afw.table.  Given the above, I think our loaders for reference catalogs must have a step that permits them to make the in-memory catalogs different from the on-disk ones, and I wonder if we should utilize that even more to allow the on-disk form to use more natural units for humans while we standardize to radians in-memory.  Ideally, we would also define an on-disk schema that could survive a switch from afw.table to something Pandas, Arrow, or Astropy-based with a loader that provides options for retaining the human-friendly units.  But perhaps I'm focusing too much on the on-disk format and long-term persistence of a format the Butler should essentially completely abstract away.

            Show
            jbosch Jim Bosch added a comment - Russell Owen , could you remind me why we have: centroid : position of star on an exposure, if relevant (an lsst.afw.Point2D) hasCentroid : is centroid usable? Are these just workspace created for algorithms that use the reference catalog?  If so, are they actually included in the on-disk representation? I have similar concerns about the various camFlux columns; mixing quantities derived from the image we're processing with those from the reference catalog seems like bad design, though I suspect it's a workaround for joins and schema manipulation of afw.tables being clunky.   More generally, I started to try to put together a counter-proposal with mostly minor changes, and I found myself quite dissatisfied with our options for dealing with units in afw.table.  Given the above, I think our loaders for reference catalogs must have a step that permits them to make the in-memory catalogs different from the on-disk ones, and I wonder if we should utilize that even more to allow the on-disk form to use more natural units for humans while we standardize to radians in-memory.  Ideally, we would also define an on-disk schema that could survive a switch from afw.table to something Pandas, Arrow, or Astropy-based with a loader that provides options for retaining the human-friendly units.  But perhaps I'm focusing too much on the on-disk format and long-term persistence of a format the Butler should essentially completely abstract away.
            Hide
            rowen Russell Owen added a comment -

            Jim Bosch yes "centroid" is a workspace. Yes it is persisted; it is debatable if that is useful but one can imagine use cases and it's easier to persist it than not.

            Show
            rowen Russell Owen added a comment - Jim Bosch yes "centroid" is a workspace. Yes it is persisted; it is debatable if that is useful but one can imagine use cases and it's easier to persist it than not.
            Hide
            rowen Russell Owen added a comment -

            Jim Bosch do you have a suggestion for bringing this to a close? Perhaps we should convene a small group to discuss it?

            Show
            rowen Russell Owen added a comment - Jim Bosch do you have a suggestion for bringing this to a close? Perhaps we should convene a small group to discuss it?
            Hide
            jbosch Jim Bosch added a comment - - edited

             
             
            I've put together a counter proposal that adds off-diagonal covariance terms (as optional columns) and cuts down on the number of flags a bit more.  Full list is here:

            https://docs.google.com/spreadsheets/d/1NsYZFUuaYfl9MMtJu8V0yxL6tG2S_IDy5iBNWuEktXI/edit?usp=sharing

            (It turns out both Jira and Confluence have terrible table support).

            A few explanations and additional rules I'd propose:

            • Grey rows are columns I'd ultimately like to get rid of, as I think we need to better separate reference catalog content from the properties of whatever image we're presumably processing when using the reference catalog, but I do not think we should try to change this now.  The only exception is that I think we should consider supporting Err instead of (or in addition to, for backwards compatibility) Sigma for flux uncertainties, if we're going to switch to Err for other uncertainties (changes highlighted in red).
            • I am otherwise trying not to mess with any of the photometry columns right now.
            • Fields never populated by a catalog should never be present in the schema.  Off-diagonal uncertainty fields should only be present when both corresponding diagonal uncertainty fields are.
            • After bashing my head against this for a while, I think we should just use radians across the board.  It's what Angle does, and the covariance units turn into a nightmare if we have any other angular units floating around.
            • The only new flags I'm proposing are parallax_flag and pm_flag, which indicate whether the values or uncertainties for that row are not trustworthy.  Absence of a column for the full catalog should be indicated instead by that column not being in the schema.  I prefer flags that indicate failure and this naming scheme because it's the precedent set by our own output catalogs.
            • Invalid per-row uncertainties should not be set to NaN without also setting the corresponding flag.  I suppose we could usually replace missing off-diagonal per-row uncertainties with zero, as that's probably often a good approximation, but I think it's unlikely this scenario will ever come up (i.e. either a catalog will always provide these uncertainties, or it never will).
            • I find the format of the off-diagonal covariance fields ugly and hard to read, but it's what CovarianceMatrixKey uses, it uses that because that's the convention we defined ages ago for the public database schema, and I have neither the time to try to change that now nor a better idea with which to replace it.
            Show
            jbosch Jim Bosch added a comment - - edited     I've put together a counter proposal that adds off-diagonal covariance terms (as optional columns) and cuts down on the number of flags a bit more.  Full list is here: https://docs.google.com/spreadsheets/d/1NsYZFUuaYfl9MMtJu8V0yxL6tG2S_IDy5iBNWuEktXI/edit?usp=sharing (It turns out both Jira and Confluence have terrible table support). A few explanations and additional rules I'd propose: Grey rows are columns I'd ultimately like to get rid of, as I think we need to better separate reference catalog content from the properties of whatever image we're presumably processing when using the reference catalog, but I do not think we should try to change this now.  The only exception is that I think we should consider supporting Err instead of (or in addition to, for backwards compatibility) Sigma for flux uncertainties, if we're going to switch to Err for other uncertainties (changes highlighted in red). I am otherwise trying not to mess with any of the photometry columns right now. Fields never populated by a catalog should never be present in the schema.  Off-diagonal uncertainty fields should only be present when both corresponding diagonal uncertainty fields are. After bashing my head against this for a while, I think we should just use radians across the board.  It's what Angle does, and the covariance units turn into a nightmare if we have any other angular units floating around. The only new flags I'm proposing are parallax_flag and pm_flag, which indicate whether the  values or uncertainties for that row are not trustworthy.  Absence of a column for the full catalog should be indicated instead by that column not being in the schema.  I prefer flags that indicate failure and this naming scheme because it's the precedent set by our own output catalogs. Invalid per-row  uncertainties should not be set to NaN without also setting the corresponding flag.  I suppose we could usually replace missing off-diagonal per-row uncertainties with zero, as that's probably often a good approximation, but I think it's unlikely this scenario will ever come up (i.e. either a catalog will always provide these uncertainties, or it never will). I find the format of the off-diagonal covariance fields ugly and hard to read, but it's what CovarianceMatrixKey uses, it uses that because that's the convention we defined ages ago for the public database schema, and I have neither the time to try to change that now nor a better idea with which to replace it.
            Hide
            rowen Russell Owen added a comment - - edited

            I am willing to adopt Jim Bosch's proposal with some comments:

            I believe we can get rid of the camFlux fields in new catalogs. A global search turns up no usage, other than 3 unit tests.

            Given that we are going to mix proper motion and position via covariance terms, we probably should use radians for proper motion, though I am not happy about it. Position is read as a SpherePoint, so the elements can be obtained in any units. Proper motion will have to be read as scalar fields, and as far as I know, we have no "angle" key, so we'll have to deal with the units the hard way.

            I will note that Gaia uses dimensionless units for terms such as pmra_pmdec_corr (see the [Gaia data model|http http://gea.esac.esa.int/archive/documentation/GDR1/datamodel/Ch1/gaia_source.html]) without any explanation that I could find (the description field says "see description", which is not helpful). I would like to understand why they do that, and whether we should emulate it, before making a final decision on units.

            Show
            rowen Russell Owen added a comment - - edited I am willing to adopt Jim Bosch 's proposal with some comments: I believe we can get rid of the camFlux fields in new catalogs. A global search turns up no usage, other than 3 unit tests. Given that we are going to mix proper motion and position via covariance terms, we probably should use radians for proper motion, though I am not happy about it. Position is read as a SpherePoint, so the elements can be obtained in any units. Proper motion will have to be read as scalar fields, and as far as I know, we have no "angle" key, so we'll have to deal with the units the hard way. I will note that Gaia uses dimensionless units for terms such as pmra_pmdec_corr (see the [Gaia data model|http http://gea.esac.esa.int/archive/documentation/GDR1/datamodel/Ch1/gaia_source.html] ) without any explanation that I could find (the description field says "see description", which is not helpful). I would like to understand why they do that, and whether we should emulate it, before making a final decision on units.
            Hide
            jbosch Jim Bosch added a comment -

            **I believe we can get rid of the camFlux fields in new catalogs. A global search turns up no usage, other than 3 unit tests.

            +1

            Proper motion will have to be read as scalar fields, and as far as I know, we have no "angle" key, so we'll have to deal with the units the hard way.

            We do have Angle fields in afw.table - it's native, not via a FunctorKey.  Also, how bad would it be to use CoordKey/SpherePoint for proper motion?  Given that float and double wouldn't encode the "/year" either, I don't think Angle (or SpherePoint) for [angle]/year would be so bad.

            I will note that Gaia uses dimensionless units for terms such as pmra_pmdec_corr [...] without any explanation that I could find

            Looks like they're recording the off-diagonal terms as correlation coefficients (a_b_corr = a_b_cov/(aErr*bErr)).  That's not a bad convention, and it might be a better one than ours, but I don't think we should have a different convention for reference catalogs vs. output catalogs.  Maybe this conversation should be moved to a new RFC?

            FWIW, the afw.table conventions are derived from these old DB conventions: https://dev.lsstcorp.org/trac/wiki/dbNamingConv.  But that's also where I got xSigma instead of xErr, which already another RFC to fix.  Gregory Dubois-Felsmann, Fritz Mueller, Colin Slater, are there updated DB naming conventions elsewhere?  Should we try to formalize this a bit better for both DB usage and afw.table usage?

            Show
            jbosch Jim Bosch added a comment - **I believe we can get rid of the camFlux fields in new catalogs. A global search turns up no usage, other than 3 unit tests. +1 Proper motion will have to be read as scalar fields, and as far as I know, we have no "angle" key, so we'll have to deal with the units the hard way. We do have Angle fields in afw.table - it's native, not via a FunctorKey.  Also, how bad would it be to use CoordKey/SpherePoint for proper motion?  Given that float and double wouldn't encode the "/year" either, I don't think Angle (or SpherePoint) for [angle] /year would be so bad. I will note that Gaia uses dimensionless units for terms such as pmra_pmdec_corr [...]  without any explanation that I could find Looks like they're recording the off-diagonal terms as correlation coefficients (a_b_corr = a_b_cov/(aErr*bErr)).  That's not a bad convention, and it might be a better one than ours, but I don't think we should have a different convention for reference catalogs vs. output catalogs.  Maybe this conversation should be moved to a new RFC? FWIW, the afw.table conventions are derived from these old DB conventions: https://dev.lsstcorp.org/trac/wiki/dbNamingConv .  But that's also where I got xSigma instead of xErr, which already another RFC to fix.  Gregory Dubois-Felsmann , Fritz Mueller , Colin Slater , are there updated DB naming conventions elsewhere?  Should we try to formalize this a bit better for both DB usage and afw.table usage?
            Hide
            tjenness Tim Jenness added a comment -

            Can someone adjust the planned end date of this RFC to something reasonable?

            Show
            tjenness Tim Jenness added a comment - Can someone adjust the planned end date of this RFC to something reasonable?
            jbosch Jim Bosch made changes -
            Planned End 09/Jan/17 6:00 PM 20/Jul/18 11:00 PM
            Hide
            rowen Russell Owen added a comment -

            We do have Angle fields in afw.table - it's native, not via a FunctorKey. Also, how bad would it be to use CoordKey/SpherePoint for proper motion? Given that float and double wouldn't encode the "/year" either, I don't think Angle (or SpherePoint) for [angle]/year would be so bad.

            Sorry I mis-spoke about Angle fields. I'm glad we have them.

            I am very uncomfortable using a SpherePoint to encode a Cartesian pair of angles due to the principle: it feels like an abuse of a class that is designed for spherical coordinates, and one that might get us into trouble if we are not careful. In practice you are certainly right that it would work – the values will always be small, so we will not run into limits. Still, I would prefer to use scalar Angle fields and not represent them as SpherePoints in memory.

            Show
            rowen Russell Owen added a comment - We do have Angle fields in afw.table - it's native, not via a FunctorKey. Also, how bad would it be to use CoordKey/SpherePoint for proper motion? Given that float and double wouldn't encode the "/year" either, I don't think Angle (or SpherePoint) for [angle] /year would be so bad. Sorry I mis-spoke about Angle fields. I'm glad we have them. I am very uncomfortable using a SpherePoint to encode a Cartesian pair of angles due to the principle: it feels like an abuse of a class that is designed for spherical coordinates, and one that might get us into trouble if we are not careful. In practice you are certainly right that it would work – the values will always be small, so we will not run into limits. Still, I would prefer to use scalar Angle fields and not represent them as SpherePoints in memory.
            Hide
            ctslater Colin Slater added a comment -

            To double check my understanding, the new proposal is to store proper motion in angular units, not coordinate units as described in Simon's RFC text? The latter would preclude the use of Angle.

            It would be really nice if we could stick with human-interpretable units on parallax and PM. mas/yr is much nicer than involving radians.

            Show
            ctslater Colin Slater added a comment - To double check my understanding, the new proposal is to store proper motion in angular units, not coordinate units as described in Simon's RFC text? The latter would preclude the use of Angle . It would be really nice if we could stick with human-interpretable units on parallax and PM. mas/yr is much nicer than involving radians.
            Hide
            rowen Russell Owen added a comment - - edited

            Colin Slater I am not sure what you mean by "angular units". The proposal is to store proper motion as radians/year (originally mas/year), using the same convention as Gaia and SDSS, but now scaling from mas to radians. If you are asking about the convention for proper motion in RA: it is the projection of the proper motion vector in the direction of increasing right ascension. Thus for a given rate of motion on the sky proper motion in RA will be the same, regardless of the declination of the star (and thus proper motion in RA is not dRA/dec).

            I think Jim Bosch is proposing to use radians for two reasons:

            • we can use an Angle key to retrieve pmra and pmdec, thus obtaining it in any units we like
            • it simplifies the units for covariance terms between position and proper motion, e.g. dec vs. pmra
            Show
            rowen Russell Owen added a comment - - edited Colin Slater I am not sure what you mean by "angular units". The proposal is to store proper motion as radians/year (originally mas/year), using the same convention as Gaia and SDSS, but now scaling from mas to radians. If you are asking about the convention for proper motion in RA: it is the projection of the proper motion vector in the direction of increasing right ascension. Thus for a given rate of motion on the sky proper motion in RA will be the same, regardless of the declination of the star (and thus proper motion in RA is not dRA/dec). I think Jim Bosch is proposing to use radians for two reasons: we can use an Angle key to retrieve pmra and pmdec, thus obtaining it in any units we like it simplifies the units for covariance terms between position and proper motion, e.g. dec vs. pmra
            Hide
            rowen Russell Owen added a comment - - edited

            Regarding units: I don't have strong feelings, but I strongly prefer that we can read the components of proper motion as instances of lsst::geom::Angle. I believe at the moment that requires them to be saved as radians/year (or any other time unit) – Jim Bosch please correct me if I'm wrong.

            In the future I hope that will not be the case and we can save both proper motion in milliarcsec/year and position in degrees and still read them in using Angle and SpherePoint.

            Show
            rowen Russell Owen added a comment - - edited Regarding units: I don't have strong feelings, but I strongly prefer that we can read the components of proper motion as instances of lsst::geom::Angle . I believe at the moment that requires them to be saved as radians/year (or any other time unit) – Jim Bosch please correct me if I'm wrong. In the future I hope that will not be the case and we can save both proper motion in milliarcsec/year and position in degrees and still read them in using Angle and SpherePoint.
            rowen Russell Owen made changes -
            Risk Score 0 1
            Hide
            jbosch Jim Bosch added a comment -

            I believe at the moment that requires them to be saved as radians/year (or any other time unit)

            Correct.

            In the future I hope that will not be the case and we can save both proper motion in milliarcsec/year and position in degrees and still read them in using Angle and SpherePoint.

            +1.  I have not given much thought to whether that would be easy to do.

            Show
            jbosch Jim Bosch added a comment - I believe at the moment that requires them to be saved as radians/year (or any other time unit) Correct. In the future I hope that will not be the case and we can save both proper motion in milliarcsec/year and position in degrees and still read them in using Angle and SpherePoint. +1.  I have not given much thought to whether that would be easy to do.
            Hide
            ctslater Colin Slater added a comment -

            It would definitely be nice to have automatic support for reading Angles in different units, but in the absence of such a feature I won't object to storing radians.

            Show
            ctslater Colin Slater added a comment - It would definitely be nice to have automatic support for reading Angles in different units, but in the absence of such a feature I won't object to storing radians.
            jbosch Jim Bosch made changes -
            Link This issue relates to DM-15174 [ DM-15174 ]
            Hide
            jbosch Jim Bosch added a comment -

            I took a look at support for different Angle units in afw.table, and documented my findings on DM-15174.  I do not have time to work on it now, but if Russell Owen feels comfortable taking that on (and it's consistent with what he's scheduled to do), it does sound like it may be worth doing before settling the schema question.

             -

            Show
            jbosch Jim Bosch added a comment - I took a look at support for different Angle units in afw.table, and documented my findings on DM-15174 .  I do not have time to work on it now, but if Russell Owen feels comfortable taking that on (and it's consistent with what he's scheduled to do), it does sound like it may be worth doing before settling the schema question.  -
            Hide
            tjenness Tim Jenness added a comment -

            Are we close to adopting this again?

            Show
            tjenness Tim Jenness added a comment - Are we close to adopting this again?
            Hide
            jbosch Jim Bosch added a comment -

            I've reassigned this to Russell Owen.  I am okay with where we landed in the last few comments; I'll let him accept if he is as well.

            Show
            jbosch Jim Bosch added a comment - I've reassigned this to Russell Owen .  I am okay with where we landed in the last few comments; I'll let him accept if he is as well.
            jbosch Jim Bosch made changes -
            Assignee Simon Krughoff [ krughoff ] Russell Owen [ rowen ]
            Hide
            rowen Russell Owen added a comment - - edited

            Adopted using Jim Bosch's proposal.

            Angle fields will be stored as radians, and proper motion as radians/year, for now. I filed RFC-502 suggesting we add support for persisting angles in other units.

            Show
            rowen Russell Owen added a comment - - edited Adopted using Jim Bosch 's proposal . Angle fields will be stored as radians, and proper motion as radians/year, for now. I filed RFC-502 suggesting we add support for persisting angles in other units.
            rowen Russell Owen made changes -
            Risk Score 1 0
            rowen Russell Owen made changes -
            Link This issue relates to RFC-502 [ RFC-502 ]
            rowen Russell Owen made changes -
            Link This issue relates to RFC-333 [ RFC-333 ]
            rowen Russell Owen made changes -
            Status Proposed [ 10805 ] Adopted [ 10806 ]
            rowen Russell Owen made changes -
            Resolution Done [ 10000 ]
            Status Adopted [ 10806 ] Implemented [ 11105 ]

              People

              • Assignee:
                rowen Russell Owen
                Reporter:
                krughoff Simon Krughoff
                Watchers:
                Christopher Waters, Colin Slater, Jim Bosch, John Parejko, Kian-Tat Lim, Paul Price, Robert Lupton, Russell Owen, Simon Krughoff, Tim Jenness
              • Votes:
                1 Vote for this issue
                Watchers:
                10 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:
                  Planned End:

                  Summary Panel