Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-32198

Set sentinel values for flag/integer columns for missing bands in Object tables

    XMLWordPrintable

    Details

      Description

      Implementation ticket for RFC-808.

      • A copy of the template dataframe will be made to maintain indexes.
      • All floating-point columns will be filled with a configurable value, default to NaN.
      • All signed integer columns will be filled with a configurable value, default to -1.
      • We do not allow unsigned integers in our Parquet tables.
      • Most flag columns will be filled with True, as most flags are "bad" flags that signify a failure of some measurement (and these blank columns are all essentially failures).
      • A specific list of "good" flag columns (goodFlags) will be specified that will default to False. Currently, in our output schema this list is ['calib_astrometry_used', 'calib_photometry_reserved', 'calib_photometry_used', 'calib_psf_candidate', 'calib_psf_used']. Not all of these columns must be present in the object table. In this way, users will never accidentally select these blank objects when looking at which objects were used for psf estimation, for example.

      Code will be adapted from https://github.com/lsst/pipe_tasks/commit/d560b2deaa52671537cb80230c648da02ad0b24b

        Attachments

          Issue Links

            Activity

            Hide
            lauren Lauren MacArthur added a comment -

            Ok, I got myself worried about some of the non-per-band columns that are booleans but technically (likely) fall into the "goodFlag" category in terms of the appropriate fill value (e.g. detect_isPrimary, sky_object, detect_isDeblendedModelSource, etc.). However, I then realized that these will all get populated appropriately precisely because they are not per-band and come from the ref catalog (phew!).

            Besides one missing "goodFlag" in the list (namely calib_psf_reserved), this looks good to me (but again with the caveat that I'm not very familiar with typing nuances...but I did run the code and what looked horrible before now looks good!)

            Thanks for getting the ball rolling on this one with the RFC and for taking care of the implementation!

            Show
            lauren Lauren MacArthur added a comment - Ok, I got myself worried about some of the non-per-band columns that are booleans but technically (likely) fall into the "goodFlag" category in terms of the appropriate fill value (e.g. detect_isPrimary , sky_object , detect_isDeblendedModelSource , etc.). However, I then realized that these will all get populated appropriately precisely because they are not per-band and come from the ref catalog (phew!). Besides one missing "goodFlag" in the list (namely calib_psf_reserved ), this looks good to me (but again with the caveat that I'm not very familiar with typing nuances...but I did run the code and what looked horrible before now looks good!) Thanks for getting the ball rolling on this one with the RFC and for taking care of the implementation!
            Hide
            lauren Lauren MacArthur added a comment -

            One comment on the PR.  I also wonder if this warrants a quick post on community (and/or an update of any other documentation where noting these sentinels would be appropriate)?

            Show
            lauren Lauren MacArthur added a comment - One comment on the PR.  I also wonder if this warrants a quick post on community (and/or an update of any other documentation where noting these sentinels would be appropriate)?
            Hide
            erykoff Eli Rykoff added a comment -

            Thank you very much for the review and catching the missing good flag! Final jenkins is https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/35197/pipeline .

            And yes, that's true, an implemented RFC should have an associated community post. I'll write that after the merge.

            Show
            erykoff Eli Rykoff added a comment - Thank you very much for the review and catching the missing good flag! Final jenkins is https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/35197/pipeline . And yes, that's true, an implemented RFC should have an associated community post. I'll write that after the merge.

              People

              Assignee:
              erykoff Eli Rykoff
              Reporter:
              erykoff Eli Rykoff
              Reviewers:
              Lauren MacArthur
              Watchers:
              Eli Rykoff, Lauren MacArthur
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.