Details
-
Type:
Story
-
Status: Done
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: pipe_tasks
-
Story Points:2
-
Epic Link:
-
Team:Data Release Production
-
Urgent?:No
Description
Implementation ticket for RFC-808.
- A copy of the template dataframe will be made to maintain indexes.
- All floating-point columns will be filled with a configurable value, default to NaN.
- All signed integer columns will be filled with a configurable value, default to -1.
- We do not allow unsigned integers in our Parquet tables.
- Most flag columns will be filled with True, as most flags are "bad" flags that signify a failure of some measurement (and these blank columns are all essentially failures).
- A specific list of "good" flag columns (goodFlags) will be specified that will default to False. Currently, in our output schema this list is ['calib_astrometry_used', 'calib_photometry_reserved', 'calib_photometry_used', 'calib_psf_candidate', 'calib_psf_used']. Not all of these columns must be present in the object table. In this way, users will never accidentally select these blank objects when looking at which objects were used for psf estimation, for example.
Code will be adapted from https://github.com/lsst/pipe_tasks/commit/d560b2deaa52671537cb80230c648da02ad0b24b
Attachments
Issue Links
- is triggered by
-
- Implemented
Activity
One comment on the PR. I also wonder if this warrants a quick post on community (and/or an update of any other documentation where noting these sentinels would be appropriate)?
Thank you very much for the review and catching the missing good flag! Final jenkins is https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/35197/pipeline .
And yes, that's true, an implemented RFC should have an associated community post. I'll write that after the merge.
Ok, I got myself worried about some of the non-per-band columns that are booleans but technically (likely) fall into the "goodFlag" category in terms of the appropriate fill value (e.g. detect_isPrimary, sky_object, detect_isDeblendedModelSource, etc.). However, I then realized that these will all get populated appropriately precisely because they are not per-band and come from the ref catalog (phew!).
Besides one missing "goodFlag" in the list (namely calib_psf_reserved), this looks good to me (but again with the caveat that I'm not very familiar with typing nuances...but I did run the code and what looked horrible before now looks good!)
Thanks for getting the ball rolling on this one with the RFC and for taking care of the implementation!