Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-22322

Add topic field descriptions and units to the Avro schema

    Details

    • Type: Improvement
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
    • Story Points:
      1
    • Sprint:
      TSSW Sprint - Nov 25 - Dec 6
    • Team:
      Telescope and Site

      Description

      Once DM-22293 is implemented topic metadata becomes available in salobj: in particular:

      • sal_version
      • xml_version
      • description and units for each field

      Also I hope we will eventually a description for each topic, but it's not in the IDL yet. See DM-22294.

      Write this metadata to the Avro schema. Where practical match the old EFD schema. See https://sqr-029.lsst.io/#id17

      At the top level add:

      • sal_version. This existed in the original EFD
      • xml_version. This is new.

      For each field add:

      • description: the description (formerly called "doc")
      • units: one guess (formerly called "sal_units")

      Angelo Fausti requests that the fields be an empty string if the data is unknown. This requires translation because salobj uses None.

        Attachments

          Issue Links

            Activity

            Hide
            rowen Russell Owen added a comment - - edited

            Pull request: https://github.com/lsst-ts/ts_salkafka/pull/6

            The new schema fields will only be present if the metadata is available. In particular the following are absent when using ts_sal 4.0 and present when using 4.1 pre-release to build the IDL files:

            • sal_version, xml_version and description (at the top level)
            • description and units for array-valued fields (at the field level)

            I also took the opportunity to modernize the unit tests to use asynctest.

            If you are willing, please check that the new schemas meet your needs.

            Show
            rowen Russell Owen added a comment - - edited Pull request: https://github.com/lsst-ts/ts_salkafka/pull/6 The new schema fields will only be present if the metadata is available. In particular the following are absent when using ts_sal 4.0 and present when using 4.1 pre-release to build the IDL files: sal_version, xml_version and description (at the top level) description and units for array-valued fields (at the field level) I also took the opportunity to modernize the unit tests to use asynctest. If you are willing, please check that the new schemas meet your needs.
            Hide
            afausti Angelo Fausti added a comment - - edited

            I'm running a SAL develop environment using:

            docker run -it lsstts/develop-env:salobj4_develop

            I've checked out tickets/DM-22293 for ts_salobj and tickets/DM-22322 for ts_salkafka and used the following commands to test the new Avro schemas:

            run_salkafka_producer.py --broker efd0.lsst.codes:9094 --registry https://registry-efd.lsst.codes/ ScriptQueue 
            

            and

            run_script_queue.py 1
            

            Here's an example of a schema with the new metadata:

            // 20191126115114
            // https://registry-efd.lsst.codes/subjects/lsst.sal.ScriptQueue.logevent_heartbeat-value/versions/4
             
            {
              "subject": "lsst.sal.ScriptQueue.logevent_heartbeat-value",
              "version": 4,
              "id": 618,
              "schema": "{\"type\":\"record\",\"name\":\"logevent_heartbeat\",\"namespace\":\"lsst.sal.ScriptQueue\",\"fields\":[{\"name\":\"private_kafkaStamp\",\"type\":\"double\",\"description\":\"TAI time at which the Kafka message was created.\",\"units\":\"second\"},{\"name\":\"ScriptQueueID\",\"type\":\"long\"},{\"name\":\"private_revCode\",\"type\":\"string\",\"description\":\"Revision code of topic\",\"units\":\"unitless\"},{\"name\":\"private_sndStamp\",\"type\":\"double\",\"description\":\"TAI at sender\",\"units\":\"secs\"},{\"name\":\"private_rcvStamp\",\"type\":\"double\",\"description\":\"TAI at receiver\",\"units\":\"secs\"},{\"name\":\"private_seqNum\",\"type\":\"long\",\"description\":\"Sequence number\",\"units\":\"unitless\"},{\"name\":\"private_origin\",\"type\":\"long\",\"description\":\"PID code of sender\",\"units\":\"unitless\"},{\"name\":\"private_host\",\"type\":\"long\",\"description\":\"IP of sender\",\"units\":\"unitless\"},{\"name\":\"heartbeat\",\"type\":\"boolean\",\"description\":\"Beats the drum.\",\"units\":\"unitless\"},{\"name\":\"priority\",\"type\":\"long\",\"description\":\"Priority code\",\"units\":\"unitless\"}],\"sal_version\":\"4.1.0\",\"xml_version\":\"1.0.0\"}"
            }
            

            Note that we have "sal_version":"4.1.0", "xml_version":"1.0.0" description and units for the fields. In particular, note that description and units are not added for the ScriptQueueID field because they are missing in the corresponding IDL.

            It looks all good.

            Show
            afausti Angelo Fausti added a comment - - edited I'm running a SAL develop environment using: docker run -it lsstts/develop-env:salobj4_develop I've checked out tickets/ DM-22293 for ts_salobj and tickets/ DM-22322 for ts_salkafka and used the following commands to test the new Avro schemas: run_salkafka_producer.py --broker efd0.lsst.codes:9094 --registry https://registry-efd.lsst.codes/ ScriptQueue and run_script_queue.py 1 Here's an example of a schema with the new metadata: // 20191126115114 // https://registry-efd.lsst.codes/subjects/lsst.sal.ScriptQueue.logevent_heartbeat-value/versions/4   { "subject": "lsst.sal.ScriptQueue.logevent_heartbeat-value", "version": 4, "id": 618, "schema": "{\"type\":\"record\",\"name\":\"logevent_heartbeat\",\"namespace\":\"lsst.sal.ScriptQueue\",\"fields\":[{\"name\":\"private_kafkaStamp\",\"type\":\"double\",\"description\":\"TAI time at which the Kafka message was created.\",\"units\":\"second\"},{\"name\":\"ScriptQueueID\",\"type\":\"long\"},{\"name\":\"private_revCode\",\"type\":\"string\",\"description\":\"Revision code of topic\",\"units\":\"unitless\"},{\"name\":\"private_sndStamp\",\"type\":\"double\",\"description\":\"TAI at sender\",\"units\":\"secs\"},{\"name\":\"private_rcvStamp\",\"type\":\"double\",\"description\":\"TAI at receiver\",\"units\":\"secs\"},{\"name\":\"private_seqNum\",\"type\":\"long\",\"description\":\"Sequence number\",\"units\":\"unitless\"},{\"name\":\"private_origin\",\"type\":\"long\",\"description\":\"PID code of sender\",\"units\":\"unitless\"},{\"name\":\"private_host\",\"type\":\"long\",\"description\":\"IP of sender\",\"units\":\"unitless\"},{\"name\":\"heartbeat\",\"type\":\"boolean\",\"description\":\"Beats the drum.\",\"units\":\"unitless\"},{\"name\":\"priority\",\"type\":\"long\",\"description\":\"Priority code\",\"units\":\"unitless\"}],\"sal_version\":\"4.1.0\",\"xml_version\":\"1.0.0\"}" } Note that we have "sal_version":"4.1.0" , "xml_version":"1.0.0" description and units for the fields. In particular, note that description and units are not added for the ScriptQueueID field because they are missing in the corresponding IDL. It looks all good.
            Hide
            afausti Angelo Fausti added a comment -

            Tests are passing locally with SAL 4.1

             
            > docker run -it lsstts/develop-env:salobj4_develop
             
            [saluser@becad4ed3bc6 ~]$ cd repos/ts_salobj/
            [saluser@becad4ed3bc6 ts_salobj]$ git pull
            .
            .
            .
            [saluser@becad4ed3bc6 ts_salobj]$ git branch
            * develop
            [saluser@becad4ed3bc6 ts_salobj]$ setup -r .
            [saluser@becad4ed3bc6 ts_salobj]$ cd ..
            [saluser@becad4ed3bc6 repos]$ git clone https://github.com/lsst-ts/ts_salkafka.git
            [saluser@becad4ed3bc6 repos]$ cd ts_salkafka/
            [saluser@becad4ed3bc6 ts_salkafka]$ git pull
            [saluser@becad4ed3bc6 ts_salkafka]$ git checkout tickets/DM-22322
            [saluser@becad4ed3bc6 ts_salkafka]$ git branch
              master
            * tickets/DM-22322
            [saluser@becad4ed3bc6 ts_salkafka]$ setup -r .
            [saluser@becad4ed3bc6 ts_salkafka]$ pytest
            
            

            Show
            afausti Angelo Fausti added a comment - Tests are passing locally with SAL 4.1   > docker run -it lsstts/develop-env:salobj4_develop   [saluser@becad4ed3bc6 ~]$ cd repos/ts_salobj/ [saluser@becad4ed3bc6 ts_salobj]$ git pull . . . [saluser@becad4ed3bc6 ts_salobj]$ git branch * develop [saluser@becad4ed3bc6 ts_salobj]$ setup -r . [saluser@becad4ed3bc6 ts_salobj]$ cd .. [saluser@becad4ed3bc6 repos]$ git clone https://github.com/lsst-ts/ts_salkafka.git [saluser@becad4ed3bc6 repos]$ cd ts_salkafka/ [saluser@becad4ed3bc6 ts_salkafka]$ git pull [saluser@becad4ed3bc6 ts_salkafka]$ git checkout tickets/DM-22322 [saluser@becad4ed3bc6 ts_salkafka]$ git branch master * tickets/DM-22322 [saluser@becad4ed3bc6 ts_salkafka]$ setup -r . [saluser@becad4ed3bc6 ts_salkafka]$ pytest
            Hide
            rowen Russell Owen added a comment -

            Merged to develop. I will tag a new release after I tag a new salobj release. I am hoping to merge DM-22269 first.

            Show
            rowen Russell Owen added a comment - Merged to develop. I will tag a new release after I tag a new salobj release. I am hoping to merge DM-22269 first.

              People

              • Assignee:
                rowen Russell Owen
                Reporter:
                rowen Russell Owen
                Reviewers:
                Angelo Fausti
                Watchers:
                Angelo Fausti, Russell Owen
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel