Details
-
Type:
Story
-
Status: Done
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: dax_obscore
-
Labels:
-
Sprint:DB_F22_6
-
Team:Data Access and Database
-
Urgent?:No
Description
For DP0.2, we need to create some additional columns during extraction, and populate them with values based on DataId attribute values. The dax_osbcore configuration facility already supports creation of new columns, but only constant data for these new columns is supported. This ticker requests extending the new columns facility to support value templating similar to what is already supported elsewhere in the configuration.
Colin Slater could probably comment here on specific columns and DataIds needed for DP0.2. See also DM-34685.
Attachments
Issue Links
- blocks
-
- Reviewed
Activity
One interesting issue is that the result of template expansion is a string (it just uses plain Python format() method). I can imagine that for some columns it may be important to use natural type when exporting to CSV/Parquet, e.g. int or float. I probably need something slightly more complicated when specifying this type of template columns. I'll try to think of something that is not too complex.
LGTM; K-T took a look as well
Merged, thanks for reviews!
To test it I used this addition to a config file:
extra_columns:
|
lsst_visit:
|
template: "{visit}"
|
type: "int"
|
lsst_detector:
|
template: "{detector}"
|
type: "int"
|
lsst_tract:
|
template: "{tract}"
|
type: "int"
|
lsst_patch:
|
template: "{patch}"
|
type: "int"
|
lsst_band:
|
template: "{band}"
|
type: "str"
|
lsst_filter:
|
template: "{physical_filter}"
|
type: "str"
|
Which covers what Colin suggested in the comment above.
My proposal is to add extra_columns for: visit, detector, tract, patch, band, filter, and to prefix each of these column names with "lsst_".