Details
-
Type:
Story
-
Status: Done
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: dax_obscore
-
Labels:
-
Sprint:DB_F22_6
-
Team:Data Access and Database
-
Urgent?:No
Description
For DP0.2, we need to create some additional columns during extraction, and populate them with values based on DataId attribute values. The dax_osbcore configuration facility already supports creation of new columns, but only constant data for these new columns is supported. This ticker requests extending the new columns facility to support value templating similar to what is already supported elsewhere in the configuration.
Colin Slater could probably comment here on specific columns and DataIds needed for DP0.2. See also DM-34685.
Attachments
Issue Links
- blocks
-
- Reviewed
Activity
| Field | Original Value | New Value |
|---|---|---|
| Summary | Extend dax_obscore extra_columns config to data id templating | Extend dax_obscore extra_columns config to support data id templating |
| Description | (Andy: filling in details per our discussion right now, but here's the ticket number anyway in case you want to start a branch while I type... :)) | For DP0.2, we need to create some additional columns during extraction, and populate them with values based on DataId attribute values. The dax_osbcore configuration facility already supports creation of new columns, but only constant data for these new columns is supported. This ticker requests extending new column support |
| Summary | Extend dax_obscore extra_columns config to support data id templating | Extend dax_obscore extra_columns to support value templating |
| Summary | Extend dax_obscore extra_columns to support value templating | Extend dax_obscore extra_columns facility to support value templating |
| Description | For DP0.2, we need to create some additional columns during extraction, and populate them with values based on DataId attribute values. The dax_osbcore configuration facility already supports creation of new columns, but only constant data for these new columns is supported. This ticker requests extending new column support |
For DP0.2, we need to create some additional columns during extraction, and populate them with values based on DataId attribute values. The dax_osbcore configuration facility already supports creation of new columns, but only constant data for these new columns is supported. This ticker requests extending the new columns facility to support value templating similar to what is already supported elsewhere in the configuration.
[~ctslater] could probably comment here on specific columns and DataIds needed for DP0.2. See also DM-34685. |
One interesting issue is that the result of template expansion is a string (it just uses plain Python format() method). I can imagine that for some columns it may be important to use natural type when exporting to CSV/Parquet, e.g. int or float. I probably need something slightly more complicated when specifying this type of template columns. I'll try to think of something that is not too complex.
| Status | To Do [ 10001 ] | In Progress [ 3 ] |
| Assignee | Andy Salnikov [ salnikov ] | Fritz Mueller [ fritzm ] |
| Assignee | Fritz Mueller [ fritzm ] | Andy Salnikov [ salnikov ] |
| Reviewers | Fritz Mueller [ fritzm ] | |
| Status | In Progress [ 3 ] | In Review [ 10004 ] |
LGTM; K-T took a look as well
| Status | In Review [ 10004 ] | Reviewed [ 10101 ] |
| Resolution | Done [ 10000 ] | |
| Status | Reviewed [ 10101 ] | Done [ 10002 ] |
Merged, thanks for reviews!
To test it I used this addition to a config file:
extra_columns:
|
lsst_visit:
|
template: "{visit}"
|
type: "int"
|
lsst_detector:
|
template: "{detector}"
|
type: "int"
|
lsst_tract:
|
template: "{tract}"
|
type: "int"
|
lsst_patch:
|
template: "{patch}"
|
type: "int"
|
lsst_band:
|
template: "{band}"
|
type: "str"
|
lsst_filter:
|
template: "{physical_filter}"
|
type: "str"
|
Which covers what Colin suggested in the comment above.
| Labels | ObsCore |
My proposal is to add extra_columns for: visit, detector, tract, patch, band, filter, and to prefix each of these column names with "lsst_".