Fix Version/s: None
Team:Data Access and Database
A need in this improvement was triggered by an experience of ingesting catalogs into Qserv as per
DM-24587. It was noticed that Qserv deployments within the Kubernetes environment exhibit extra failure modes which are rarely seen within the "traditional" host-based static setups. This includes more frequent DNS failures (as Kubernetes has its own DNS server), more frequent connection losses to overloaded services, worker pods being restarted (and disappearing from the Kubernetes DNS), etc. All together this brings in a new spectrum of issues which may need to be handled by the clients workflows.
Here is some background on this subject. When the current Ingest system was being designed one of the main concerns to be addressed were failures seen when loading table contributions. That appeared as the highest risk area due to the large amount of data sent to servers as well as a high frequency of communications. And for that, the super-transactions provide a reasonable protection. For instance, if a client loses a connection with a worker’s ingest server after a successful completion of some file ingest operation, and the client gets a connection error then it can always retry the “failed” operation within a new transaction. This is some kind of “false positive” scenario, which is generally harmless. When it comes to connection failures to the Master Replication Controller, then it all depends on a specific operation, and on its context. Some operations are safe to repeat. Others may require extra interactions with the Controller to figure out a state of a catalog. One example is the transaction management. If one sees a connection failure while waiting for a response from the REST service on a new transaction request it may not mean that the operation has failed per se. The transaction could have been successfully initiated. In this case a client would have to scan existing transactions to see which transactions are active and figure out which one (if any) was attempted by the client. Unfortunately, transactions don’t have an “owner”. All a client may learn from the Ingest system is that the name of a database associated with the transaction, a state of the transaction and the start/stop timestamps of the transaction. Based on these attributes of a transaction, it would be hard to figure out which one was initiated by a specific client. Altogether this may complicated a recovery process.
This development offers a solution to the last problem by allowing a transaction-requesting client to specify an optional context for a new transaction. It would be up to the client to formulate the context and send it to the Controller along with the transaction opening request. The context would be stored in the Controller’s database along with other info on the transaction. The context could be retrieved along with the transaction records via the earlier discussed service GET /ingest/trans?database=<name>. To make things more convenient here the context could be specified as a JSON object sent with POST /ingest/trans. And it would be up to the client what to put into that object (though, it doesn’t have to be huge). As an example, the context could include a host from which the transaction was requested, the PID of a process requested the transaction, or anything else. Some ingest workflows may even generate a UUID to be sent in the context so that the the workflow could implement a proper bookkeeping for this type of operations.
This development requires:
- extending the table schema for qservReplica.transaction
- extending a schema of a JSON object send with the POST request to /ingest/trans with an optional dictionary-type attribute context.
- extending an implementation of the GET request to /ingest/trans to return a context in attribute context for each transaction reported by the service.
- updating the documentation in the Ingest System: https://confluence.lsstcorp.org/pages/viewpage.action?pageId=133333850
- (possibly) increasing the version number reported by the GET service /meta/version.