Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-35301

Extend dax_obscore extra_columns facility to support value templating

    XMLWordPrintable

Details

    • Story
    • Status: Done
    • Resolution: Done
    • None
    • dax_obscore
    • DB_F22_6
    • Data Access and Database
    • No

    Description

      For DP0.2, we need to create some additional columns during extraction, and populate them with values based on DataId attribute values.  The dax_osbcore configuration facility already supports creation of new columns, but only constant data for these new columns is supported.  This ticker requests extending the new columns facility to support value templating similar to what is already supported elsewhere in the configuration.

      ctslater could probably comment here on specific columns and DataIds needed for DP0.2.  See also DM-34685.

      Attachments

        Issue Links

          Activity

            No builds found.
            fritzm Fritz Mueller created issue -
            fritzm Fritz Mueller made changes -
            Field Original Value New Value
            Summary Extend dax_obscore extra_columns config to data id templating Extend dax_obscore extra_columns config to support data id templating
            fritzm Fritz Mueller made changes -
            Description (Andy: filling in details per our discussion right now, but here's the ticket number anyway in case you want to start a branch while I type... :)) For DP0.2, we need to create some additional columns during extraction, and populate them with values based on DataId attribute values.  The dax_osbcore configuration facility already supports creation of new columns, but only constant data for these new columns is supported.  This ticker requests extending new column support 
            fritzm Fritz Mueller made changes -
            Summary Extend dax_obscore extra_columns config to support data id templating Extend dax_obscore extra_columns to support value templating
            fritzm Fritz Mueller made changes -
            Summary Extend dax_obscore extra_columns to support value templating Extend dax_obscore extra_columns facility to support value templating
            fritzm Fritz Mueller made changes -
            Description For DP0.2, we need to create some additional columns during extraction, and populate them with values based on DataId attribute values.  The dax_osbcore configuration facility already supports creation of new columns, but only constant data for these new columns is supported.  This ticker requests extending new column support  For DP0.2, we need to create some additional columns during extraction, and populate them with values based on DataId attribute values.  The dax_osbcore configuration facility already supports creation of new columns, but only constant data for these new columns is supported.  This ticker requests extending the new columns facility to support value templating similar to what is already supported elsewhere in the configuration.

            [~ctslater] could probably comment here on specific columns and DataIds needed for DP0.2.  See also DM-34685.
            fritzm Fritz Mueller made changes -
            Link This issue blocks DM-34685 [ DM-34685 ]

            My proposal is to add extra_columns for: visit, detector, tract, patch, band, filter, and to prefix each of these column names with "lsst_".

             

            ctslater Colin Slater added a comment - My proposal is to add extra_columns for: visit, detector, tract, patch, band, filter, and to prefix each of these column names with "lsst_".  

            One interesting issue is that the result of template expansion is a string (it just uses plain Python format() method). I can imagine that for some columns it may be important to use natural type when exporting to CSV/Parquet, e.g. int or float. I probably need something slightly more complicated when specifying this type of template columns. I'll try to think of something that is not too complex.

            salnikov Andy Salnikov added a comment - One interesting issue is that the result of template expansion is a string (it just uses plain Python format() method). I can imagine that for some columns it may be important to use natural type when exporting to CSV/Parquet, e.g. int or float. I probably need something slightly more complicated when specifying this type of template columns. I'll try to think of something that is not too complex.
            fritzm Fritz Mueller made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            fritzm Fritz Mueller made changes -
            Assignee Andy Salnikov [ salnikov ] Fritz Mueller [ fritzm ]
            fritzm Fritz Mueller made changes -
            Assignee Fritz Mueller [ fritzm ] Andy Salnikov [ salnikov ]
            fritzm Fritz Mueller made changes -
            Reviewers Fritz Mueller [ fritzm ]
            Status In Progress [ 3 ] In Review [ 10004 ]

            LGTM; K-T took a look as well

            fritzm Fritz Mueller added a comment - LGTM; K-T took a look as well
            fritzm Fritz Mueller made changes -
            Status In Review [ 10004 ] Reviewed [ 10101 ]
            salnikov Andy Salnikov made changes -
            Resolution Done [ 10000 ]
            Status Reviewed [ 10101 ] Done [ 10002 ]
            salnikov Andy Salnikov added a comment - - edited

            Merged, thanks for reviews!

            To test it I used this addition to a config file:

            extra_columns:
              lsst_visit:
                template: "{visit}"
                type: "int"
              lsst_detector:
                template: "{detector}"
                type: "int"
              lsst_tract:
                template: "{tract}"
                type: "int"
              lsst_patch:
                template: "{patch}"
                type: "int"
              lsst_band:
                template: "{band}"
                type: "str"
              lsst_filter:
                template: "{physical_filter}"
                type: "str"
            

            Which covers what Colin suggested in the comment above.

            salnikov Andy Salnikov added a comment - - edited Merged, thanks for reviews! To test it I used this addition to a config file: extra_columns: lsst_visit: template: "{visit}" type: "int" lsst_detector: template: "{detector}" type: "int" lsst_tract: template: "{tract}" type: "int" lsst_patch: template: "{patch}" type: "int" lsst_band: template: "{band}" type: "str" lsst_filter: template: "{physical_filter}" type: "str" Which covers what Colin suggested in the comment above.
            salnikov Andy Salnikov made changes -
            Labels ObsCore

            People

              salnikov Andy Salnikov
              fritzm Fritz Mueller
              Fritz Mueller
              Andy Salnikov, Colin Slater, Fritz Mueller, Frossie Economou, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.