Details
-
Type:
Improvement
-
Status: Done
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: None
-
Labels:
-
Story Points:1
-
Epic Link:
-
Sprint:TSSW Sprint - Nov 25 - Dec 6
-
Team:Telescope and Site
Description
Once DM-22293 is implemented topic metadata becomes available in salobj: in particular:
- sal_version
- xml_version
- description and units for each field
Also I hope we will eventually a description for each topic, but it's not in the IDL yet. See DM-22294.
Write this metadata to the Avro schema. Where practical match the old EFD schema. See https://sqr-029.lsst.io/#id17
At the top level add:
- sal_version. This existed in the original EFD
- xml_version. This is new.
For each field add:
- description: the description (formerly called "doc")
- units: one guess (formerly called "sal_units")
Angelo Fausti requests that the fields be an empty string if the data is unknown. This requires translation because salobj uses None.
Attachments
Issue Links
- is triggered by
-
- Done
Activity
I'm running a SAL develop environment using:
docker run -it lsstts/develop-env:salobj4_develop
I've checked out tickets/DM-22293 for ts_salobj and tickets/DM-22322 for ts_salkafka and used the following commands to test the new Avro schemas:
run_salkafka_producer.py --broker efd0.lsst.codes:9094 --registry https://registry-efd.lsst.codes/ ScriptQueue
|
and
run_script_queue.py 1
|
Here's an example of a schema with the new metadata:
// 20191126115114
|
// https://registry-efd.lsst.codes/subjects/lsst.sal.ScriptQueue.logevent_heartbeat-value/versions/4
|
|
|
{
|
"subject": "lsst.sal.ScriptQueue.logevent_heartbeat-value",
|
"version": 4,
|
"id": 618,
|
"schema": "{\"type\":\"record\",\"name\":\"logevent_heartbeat\",\"namespace\":\"lsst.sal.ScriptQueue\",\"fields\":[{\"name\":\"private_kafkaStamp\",\"type\":\"double\",\"description\":\"TAI time at which the Kafka message was created.\",\"units\":\"second\"},{\"name\":\"ScriptQueueID\",\"type\":\"long\"},{\"name\":\"private_revCode\",\"type\":\"string\",\"description\":\"Revision code of topic\",\"units\":\"unitless\"},{\"name\":\"private_sndStamp\",\"type\":\"double\",\"description\":\"TAI at sender\",\"units\":\"secs\"},{\"name\":\"private_rcvStamp\",\"type\":\"double\",\"description\":\"TAI at receiver\",\"units\":\"secs\"},{\"name\":\"private_seqNum\",\"type\":\"long\",\"description\":\"Sequence number\",\"units\":\"unitless\"},{\"name\":\"private_origin\",\"type\":\"long\",\"description\":\"PID code of sender\",\"units\":\"unitless\"},{\"name\":\"private_host\",\"type\":\"long\",\"description\":\"IP of sender\",\"units\":\"unitless\"},{\"name\":\"heartbeat\",\"type\":\"boolean\",\"description\":\"Beats the drum.\",\"units\":\"unitless\"},{\"name\":\"priority\",\"type\":\"long\",\"description\":\"Priority code\",\"units\":\"unitless\"}],\"sal_version\":\"4.1.0\",\"xml_version\":\"1.0.0\"}"
|
}
|
Note that we have "sal_version":"4.1.0", "xml_version":"1.0.0" description and units for the fields. In particular, note that description and units are not added for the ScriptQueueID field because they are missing in the corresponding IDL.
It looks all good.
Tests are passing locally with SAL 4.1
|
|
> docker run -it lsstts/develop-env:salobj4_develop
|
|
|
[saluser@becad4ed3bc6 ~]$ cd repos/ts_salobj/
|
[saluser@becad4ed3bc6 ts_salobj]$ git pull
|
.
|
.
|
.
|
[saluser@becad4ed3bc6 ts_salobj]$ git branch
|
* develop
|
[saluser@becad4ed3bc6 ts_salobj]$ setup -r .
|
[saluser@becad4ed3bc6 ts_salobj]$ cd ..
|
[saluser@becad4ed3bc6 repos]$ git clone https://github.com/lsst-ts/ts_salkafka.git
|
[saluser@becad4ed3bc6 repos]$ cd ts_salkafka/
|
[saluser@becad4ed3bc6 ts_salkafka]$ git pull
|
[saluser@becad4ed3bc6 ts_salkafka]$ git checkout tickets/DM-22322
|
[saluser@becad4ed3bc6 ts_salkafka]$ git branch
|
master
|
* tickets/DM-22322
|
[saluser@becad4ed3bc6 ts_salkafka]$ setup -r .
|
[saluser@becad4ed3bc6 ts_salkafka]$ pytest
|
|
Merged to develop. I will tag a new release after I tag a new salobj release. I am hoping to merge DM-22269 first.
Pull request: https://github.com/lsst-ts/ts_salkafka/pull/6
The new schema fields will only be present if the metadata is available. In particular the following are absent when using ts_sal 4.0 and present when using 4.1 pre-release to build the IDL files:
I also took the opportunity to modernize the unit tests to use asynctest.
If you are willing, please check that the new schemas meet your needs.