XMLWordPrintable

#### Details

• Type: Improvement
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s:
• Labels:
None
• Story Points:
8
• Sprint:
DB_F20_09, DB_S21_12
• Team:
Data Access and Database

#### Description

A goal of the effort is to add persistent bookkeeping records for table contributions made during catalog ingests. This would require creating a database table to store one record per each contribution. Each record would have:

• the start time of the operation (when a request was received from a client)
• the end time of the operation (when a request was accomplished or failed)
• a status of the operation: IN-PROGRESS, SUCCESS, FAILED
• (optional) an error code should the operation failed (MySQL error, etc.)
• an identifier of a transaction (FK to the transaction table)
• the name of the catalog (a part of the composite FK to column config_database_table.database)
• the base name of a table (a part of the composite FK to column config_database_table.table)
• the name of a worker
• the chunk number (or NULL )
• the overlap flag (or NULL)
• the number of rows ingested (if the operation succeeded)
• the number of bytes ingested (if the operation succeeded)
• (optionally, if known) a source of the contribution

In the initial implementation (of the feature), the "source" name would be encoded into as:

 file:/// 

Where, values of <host>> and <absolute-path> would be reported to the Ingest service by the client application qserv-replica-file-ingest.
Later on, when the ingest service will be enchased to allow pulling table contributions over the HTTP protocol, other forms of the "source" names would be added. For example:

 http:/// 

The corresponding REST service will be added to the Master Replication Controller to pull the bookkeeping records from the table and return them to a client.

In a separate effort, the Qserv Dashboard Web application will be also extended to pull the bookkeeping records in order to display progress on the catalog ingests.

Other actions:

• update documentation on the extended REST services

#### Activity

Igor Gaponenko created issue -
Field Original Value New Value
Epic Link DM-25251 [ 435594 ]
 Description A goal of the effort is to add a persistent bookkeeping for table contributions made during catalog ingests. This would require creating a database table to store one record per each contribution. Each record would have: * the start time of the operation (when a request was received from a client) * the end time of the operation (when a request was accomplished or failed) * a completion status of the operation * (optional) an error code should the operation failed (ot enough storage, MySQL error, etc.) * an identifier of a transaction * the name of catalog * the base name of a table * (optionally) the chunk number (or {{NULL}}) * (optionally) the {{overlap}} flag (or {{NULL}}) * the number of rows ingested (if the operation succeeded) * (optionally, if known) a source of the contribution In the initial implementation (of the feature), the "source" name would be encoded into as: {code} file://<host>/<absolute-path> {code} Where, values of {{>}} and {{}} would be reported to the Ingest service by the client application {{qserv-replica-file-ingest}}. A goal of the effort is to add a persistent bookkeeping for table contributions made during catalog ingests. This would require creating a database table to store one record per each contribution. Each record would have: * the start time of the operation (when a request was received from a client) * the end time of the operation (when a request was accomplished or failed) * a completion status of the operation * (optional) an error code should the operation failed (ot enough storage, MySQL error, etc.) * an identifier of a transaction * the name of catalog * the base name of a table * (optionally) the chunk number (or {{NULL}}) * (optionally) the {{overlap}} flag (or {{NULL}}) * the number of rows ingested (if the operation succeeded) * (optionally, if known) a source of the contribution In the initial implementation (of the feature), the "source" name would be encoded into as: {code} file://<host>/<absolute-path> {code} Where, values of {{>}} and {{}} would be reported to the Ingest service by the client application {{qserv-replica-file-ingest}}. Later on, when the ingest service will be enchased to allow pulling table contributions over the HTTP protocol, other forms of the "source" names would be added. For example: {code} http://<host>/<path> {code} The corresponding REST service will be added to the Master Replication Controller to pull the bookkeeping records from the table and return to a client. In a separate effort, the Qserv Dashboard Web application will be also extended to pull the bookkeeping records in order to display a progress on the catalog ingests.
 Sprint DB_F20_06 [ 1026 ] DB_F20_09 [ 1048 ]
 Description A goal of the effort is to add a persistent bookkeeping for table contributions made during catalog ingests. This would require creating a database table to store one record per each contribution. Each record would have: * the start time of the operation (when a request was received from a client) * the end time of the operation (when a request was accomplished or failed) * a completion status of the operation * (optional) an error code should the operation failed (ot enough storage, MySQL error, etc.) * an identifier of a transaction * the name of catalog * the base name of a table * (optionally) the chunk number (or {{NULL}}) * (optionally) the {{overlap}} flag (or {{NULL}}) * the number of rows ingested (if the operation succeeded) * (optionally, if known) a source of the contribution In the initial implementation (of the feature), the "source" name would be encoded into as: {code} file://<host>/<absolute-path> {code} Where, values of {{>}} and {{}} would be reported to the Ingest service by the client application {{qserv-replica-file-ingest}}. Later on, when the ingest service will be enchased to allow pulling table contributions over the HTTP protocol, other forms of the "source" names would be added. For example: {code} http://<host>/<path> {code} The corresponding REST service will be added to the Master Replication Controller to pull the bookkeeping records from the table and return to a client. In a separate effort, the Qserv Dashboard Web application will be also extended to pull the bookkeeping records in order to display a progress on the catalog ingests. A goal of the effort is to add persistent bookkeeping for table contributions made during catalog ingests. This would require creating a database table to store one record per each contribution. Each record would have: * the start time of the operation (when a request was received from a client) * the end time of the operation (when a request was accomplished or failed) * a completion status of the operation: {{IN-PROGRESS}}, {{SUCCESS}}, {{FAILED}} * (optional) an error code should the operation failed (MySQL error, etc.) * an identifier of a transaction (FK to the {{transaction}} table) * the name of the catalog (a part of the composite FK to column {{config_database_table.database}}) * the base name of a table (a part of the composite FK to column {{config_database_table.table}}) * the name of a worker * the chunk number (or {{NULL}} ) * the {{overlap}} flag (or {{NULL}}) * the number of rows ingested (if the operation succeeded) * the number of bytes ingested (if the operation succeeded) * (optionally, if known) a source of the contribution In the initial implementation (of the feature), the "source" name would be encoded into as: {code} file://<host>/<absolute-path> {code} Where, values of {{>}} and {{}} would be reported to the Ingest service by the client application {{qserv-replica-file-ingest}}. Later on, when the ingest service will be enchased to allow pulling table contributions over the HTTP protocol, other forms of the "source" names would be added. For example: {code} http://<host>/<path> {code} The corresponding REST service will be added to the Master Replication Controller to pull the bookkeeping records from the table and return them to a client. In a separate effort, the Qserv Dashboard Web application will be also extended to pull the bookkeeping records in order to display progress on the catalog ingests.
 Description A goal of the effort is to add persistent bookkeeping for table contributions made during catalog ingests. This would require creating a database table to store one record per each contribution. Each record would have: * the start time of the operation (when a request was received from a client) * the end time of the operation (when a request was accomplished or failed) * a completion status of the operation: {{IN-PROGRESS}}, {{SUCCESS}}, {{FAILED}} * (optional) an error code should the operation failed (MySQL error, etc.) * an identifier of a transaction (FK to the {{transaction}} table) * the name of the catalog (a part of the composite FK to column {{config_database_table.database}}) * the base name of a table (a part of the composite FK to column {{config_database_table.table}}) * the name of a worker * the chunk number (or {{NULL}} ) * the {{overlap}} flag (or {{NULL}}) * the number of rows ingested (if the operation succeeded) * the number of bytes ingested (if the operation succeeded) * (optionally, if known) a source of the contribution In the initial implementation (of the feature), the "source" name would be encoded into as: {code} file://<host>/<absolute-path> {code} Where, values of {{>}} and {{}} would be reported to the Ingest service by the client application {{qserv-replica-file-ingest}}. Later on, when the ingest service will be enchased to allow pulling table contributions over the HTTP protocol, other forms of the "source" names would be added. For example: {code} http://<host>/<path> {code} The corresponding REST service will be added to the Master Replication Controller to pull the bookkeeping records from the table and return them to a client. In a separate effort, the Qserv Dashboard Web application will be also extended to pull the bookkeeping records in order to display progress on the catalog ingests. A goal of the effort is to add persistent bookkeeping records for table contributions made during catalog ingests. This would require creating a database table to store one record per each contribution. Each record would have: * the start time of the operation (when a request was received from a client) * the end time of the operation (when a request was accomplished or failed) * a completion status of the operation: {{IN-PROGRESS}}, {{SUCCESS}}, {{FAILED}} * (optional) an error code should the operation failed (MySQL error, etc.) * an identifier of a transaction (FK to the {{transaction}} table) * the name of the catalog (a part of the composite FK to column {{config_database_table.database}}) * the base name of a table (a part of the composite FK to column {{config_database_table.table}}) * the name of a worker * the chunk number (or {{NULL}} ) * the {{overlap}} flag (or {{NULL}}) * the number of rows ingested (if the operation succeeded) * the number of bytes ingested (if the operation succeeded) * (optionally, if known) a source of the contribution In the initial implementation (of the feature), the "source" name would be encoded into as: {code} file://<host>/<absolute-path> {code} Where, values of {{>}} and {{}} would be reported to the Ingest service by the client application {{qserv-replica-file-ingest}}. Later on, when the ingest service will be enchased to allow pulling table contributions over the HTTP protocol, other forms of the "source" names would be added. For example: {code} http://<host>/<path> {code} The corresponding REST service will be added to the Master Replication Controller to pull the bookkeeping records from the table and return them to a client. In a separate effort, the Qserv Dashboard Web application will be also extended to pull the bookkeeping records in order to display progress on the catalog ingests.
 Description A goal of the effort is to add persistent bookkeeping records for table contributions made during catalog ingests. This would require creating a database table to store one record per each contribution. Each record would have: * the start time of the operation (when a request was received from a client) * the end time of the operation (when a request was accomplished or failed) * a completion status of the operation: {{IN-PROGRESS}}, {{SUCCESS}}, {{FAILED}} * (optional) an error code should the operation failed (MySQL error, etc.) * an identifier of a transaction (FK to the {{transaction}} table) * the name of the catalog (a part of the composite FK to column {{config_database_table.database}}) * the base name of a table (a part of the composite FK to column {{config_database_table.table}}) * the name of a worker * the chunk number (or {{NULL}} ) * the {{overlap}} flag (or {{NULL}}) * the number of rows ingested (if the operation succeeded) * the number of bytes ingested (if the operation succeeded) * (optionally, if known) a source of the contribution In the initial implementation (of the feature), the "source" name would be encoded into as: {code} file://<host>/<absolute-path> {code} Where, values of {{>}} and {{}} would be reported to the Ingest service by the client application {{qserv-replica-file-ingest}}. Later on, when the ingest service will be enchased to allow pulling table contributions over the HTTP protocol, other forms of the "source" names would be added. For example: {code} http://<host>/<path> {code} The corresponding REST service will be added to the Master Replication Controller to pull the bookkeeping records from the table and return them to a client. In a separate effort, the Qserv Dashboard Web application will be also extended to pull the bookkeeping records in order to display progress on the catalog ingests. A goal of the effort is to add persistent bookkeeping records for table contributions made during catalog ingests. This would require creating a database table to store one record per each contribution. Each record would have: * the start time of the operation (when a request was received from a client) * the end time of the operation (when a request was accomplished or failed) * a status of the operation: {{IN-PROGRESS}}, {{SUCCESS}}, {{FAILED}} * (optional) an error code should the operation failed (MySQL error, etc.) * an identifier of a transaction (FK to the {{transaction}} table) * the name of the catalog (a part of the composite FK to column {{config_database_table.database}}) * the base name of a table (a part of the composite FK to column {{config_database_table.table}}) * the name of a worker * the chunk number (or {{NULL}} ) * the {{overlap}} flag (or {{NULL}}) * the number of rows ingested (if the operation succeeded) * the number of bytes ingested (if the operation succeeded) * (optionally, if known) a source of the contribution In the initial implementation (of the feature), the "source" name would be encoded into as: {code} file://<host>/<absolute-path> {code} Where, values of {{>}} and {{}} would be reported to the Ingest service by the client application {{qserv-replica-file-ingest}}. Later on, when the ingest service will be enchased to allow pulling table contributions over the HTTP protocol, other forms of the "source" names would be added. For example: {code} http://<host>/<path> {code} The corresponding REST service will be added to the Master Replication Controller to pull the bookkeeping records from the table and return them to a client. In a separate effort, the Qserv Dashboard Web application will be also extended to pull the bookkeeping records in order to display progress on the catalog ingests.
 Epic Link DM-25251 [ 435594 ] DM-27790 [ 442306 ]
 Sprint DB_F20_09 [ 1048 ] DB_F20_09, DB_S21_12 [ 1048, 1065 ]
 Status To Do [ 10001 ] In Progress [ 3 ]
 Story Points 2 6
 Story Points 6 8
 Link This issue blocks DM-19630 [ DM-19630 ]
 Reviewers John Gates, Nate Pease [ jgates, npease ]
 Status In Progress [ 3 ] In Review [ 10004 ]
 Reviewers John Gates, Nate Pease [ jgates, npease ] Nate Pease [ npease ]
 Link This issue blocks DM-28603 [ DM-28603 ]
 Status In Review [ 10004 ] Reviewed [ 10101 ]
 Description A goal of the effort is to add persistent bookkeeping records for table contributions made during catalog ingests. This would require creating a database table to store one record per each contribution. Each record would have: * the start time of the operation (when a request was received from a client) * the end time of the operation (when a request was accomplished or failed) * a status of the operation: {{IN-PROGRESS}}, {{SUCCESS}}, {{FAILED}} * (optional) an error code should the operation failed (MySQL error, etc.) * an identifier of a transaction (FK to the {{transaction}} table) * the name of the catalog (a part of the composite FK to column {{config_database_table.database}}) * the base name of a table (a part of the composite FK to column {{config_database_table.table}}) * the name of a worker * the chunk number (or {{NULL}} ) * the {{overlap}} flag (or {{NULL}}) * the number of rows ingested (if the operation succeeded) * the number of bytes ingested (if the operation succeeded) * (optionally, if known) a source of the contribution In the initial implementation (of the feature), the "source" name would be encoded into as: {code} file://<host>/<absolute-path> {code} Where, values of {{>}} and {{}} would be reported to the Ingest service by the client application {{qserv-replica-file-ingest}}. Later on, when the ingest service will be enchased to allow pulling table contributions over the HTTP protocol, other forms of the "source" names would be added. For example: {code} http://<host>/<path> {code} The corresponding REST service will be added to the Master Replication Controller to pull the bookkeeping records from the table and return them to a client. In a separate effort, the Qserv Dashboard Web application will be also extended to pull the bookkeeping records in order to display progress on the catalog ingests. A goal of the effort is to add persistent bookkeeping records for table contributions made during catalog ingests. This would require creating a database table to store one record per each contribution. Each record would have: * the start time of the operation (when a request was received from a client) * the end time of the operation (when a request was accomplished or failed) * a status of the operation: {{IN-PROGRESS}}, {{SUCCESS}}, {{FAILED}} * (optional) an error code should the operation failed (MySQL error, etc.) * an identifier of a transaction (FK to the {{transaction}} table) * the name of the catalog (a part of the composite FK to column {{config_database_table.database}}) * the base name of a table (a part of the composite FK to column {{config_database_table.table}}) * the name of a worker * the chunk number (or {{NULL}} ) * the {{overlap}} flag (or {{NULL}}) * the number of rows ingested (if the operation succeeded) * the number of bytes ingested (if the operation succeeded) * (optionally, if known) a source of the contribution In the initial implementation (of the feature), the "source" name would be encoded into as: {code} file://<host>/<absolute-path> {code} Where, values of {{>}} and {{}} would be reported to the Ingest service by the client application {{qserv-replica-file-ingest}}. Later on, when the ingest service will be enchased to allow pulling table contributions over the HTTP protocol, other forms of the "source" names would be added. For example: {code} http://<host>/<path> {code} The corresponding REST service will be added to the Master Replication Controller to pull the bookkeeping records from the table and return them to a client. In a separate effort, the Qserv Dashboard Web application will be also extended to pull the bookkeeping records in order to display progress on the catalog ingests. Other actions: * update documentation on the extended REST services
 Resolution Done [ 10000 ] Status Reviewed [ 10101 ] Done [ 10002 ]
 Link This issue is triggering DM-30095 [ DM-30095 ]

#### People

Assignee:
Igor Gaponenko
Reporter:
Igor Gaponenko
Reviewers:
Nate Pease
Watchers:
Fritz Mueller, Igor Gaponenko, John Gates, Nate Pease