Russell Owen that's a very interesting case. I agree, some breaking changes will have to go through - that seems inevitable. The importance of the schema compatibility checks is that we'll know that in advance. The situation now is that we only know when it breaks in production and that’s what we want to avoid. My proposal above is to enable these checks in the T&S CI so we can plan ahead and make any manual changes in Kafka or in the InfluxDB schema in preparation for that.

Tiago Ribeiro yes, if you rename a component you'll get new topics in Kafka and InfluxDB, however, the old topics won't be automatically renamed. It would require manual intervention in Kafka to delete the old topics and in the InfluxDB schema to rename the topics -that’s why we say it “breaks the schema evolution” (the old topics still exist in the EFD but they don’t exist anymore in ts_xml). Now, if we want to rename a component that’s ok, we just need to make these manual changes in advance in preparation for the deployment of the new ts_xml version.

The schema compatibility checks will add more control to these changes and detect incompatible changes early on.
The same is true for the type changes - but it is not that they are not allowed - it just that the change won’t be applied automatically to the existing schema in Kafka and in InfluxDB.

Alright so we (me, Angelo Fausti, Simon Krughoff) met with Tony Johnson Russell Owen Michael Reuter and Tiago Ribeiro to clarify a few things and discuss with more bandwidth.

Summary:

• We all agree on the problem statement.
• From the Camera point of view, it sends to the EFD telemetry and configuration. Telemetry is not special for the purposes of this ticket and is included in the discussion below. Configuration has a lot of active schema churn but is also currently of mostly transient value, so we think that we don't have a huge issue with schema non-continuity now, and at some point it will stablise and this problem will go away. The main issue here is that variable names are auto-generated and hence we cannot do the "just rename the variable if you are going to change the type on us" trick.
• In the general case, the challenge here is to detect schema changes that are not forward compatible ahead of time so they can be resolved (if accidental) or addressed in a planned way if intentional
• SQuaRE (Angelo) will work with T&S (Tiago) to build in integration tests in the T&S build chain to detect schema breakages before deployment
• Michael will describe (seperately below) the process by which these integration tests will become part of the T&S software release lifecycle
• I will schedule some time for this work in the later part of the summer cycle, as responding to this problem is going to be an even bigger headache once we have data going in the consolidated EFD (postgres) and yet another schema to manage. 
• All of the above will do a lot to mitigate the problems we have seen going forward. Meanwhile we accumulated a lot of historical cruft in the EFD and need to do a spring clean once we have stopped making the problem worse.

Thanks all, good talk. 

We will integrate a job to determine the schema changes as part of our cycle (release) build process. The schema job will be launched after the base artifact build that makes all of our SAL libraries completes. The report generated from the job will be delivered to SQuaRE so they can plan appropriately for the changes that will show up in deployment. Once we have established this job within the cycle build context, we will make a similar job for our daily and bleed builds that will allow the SAL/XML collective to decide on actions long before changes arrive at SQuaREs doorstep.

Still relevant but must be planed for another epic. Moving to my backlog epic for now.