Fix Version/s: None
Team:Data Access and Database
A goal of the effort is to add persistent bookkeeping records for table contributions made during catalog ingests. This would require creating a database table to store one record per each contribution. Each record would have:
- the start time of the operation (when a request was received from a client)
- the end time of the operation (when a request was accomplished or failed)
- a status of the operation: IN-PROGRESS, SUCCESS, FAILED
- (optional) an error code should the operation failed (MySQL error, etc.)
- an identifier of a transaction (FK to the transaction table)
- the name of the catalog (a part of the composite FK to column config_database_table.database)
- the base name of a table (a part of the composite FK to column config_database_table.table)
- the name of a worker
- the chunk number (or NULL )
- the overlap flag (or NULL)
- the number of rows ingested (if the operation succeeded)
- the number of bytes ingested (if the operation succeeded)
- (optionally, if known) a source of the contribution
In the initial implementation (of the feature), the "source" name would be encoded into as:
Where, values of <host>> and <absolute-path> would be reported to the Ingest service by the client application qserv-replica-file-ingest.
Later on, when the ingest service will be enchased to allow pulling table contributions over the HTTP protocol, other forms of the "source" names would be added. For example:
The corresponding REST service will be added to the Master Replication Controller to pull the bookkeeping records from the table and return them to a client.
In a separate effort, the Qserv Dashboard Web application will be also extended to pull the bookkeeping records in order to display progress on the catalog ingests.
- update documentation on the extended REST services