Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-26101

Add bookkeeping for table contributions requested during catalog ingests

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: Qserv
    • Labels:
      None

      Description

      A goal of the effort is to add persistent bookkeeping records for table contributions made during catalog ingests. This would require creating a database table to store one record per each contribution. Each record would have:

      • the start time of the operation (when a request was received from a client)
      • the end time of the operation (when a request was accomplished or failed)
      • a status of the operation: IN-PROGRESS, SUCCESS, FAILED
      • (optional) an error code should the operation failed (MySQL error, etc.)
      • an identifier of a transaction (FK to the transaction table)
      • the name of the catalog (a part of the composite FK to column config_database_table.database)
      • the base name of a table (a part of the composite FK to column config_database_table.table)
      • the name of a worker
      • the chunk number (or NULL )
      • the overlap flag (or NULL)
      • the number of rows ingested (if the operation succeeded)
      • the number of bytes ingested (if the operation succeeded)
      • (optionally, if known) a source of the contribution

      In the initial implementation (of the feature), the "source" name would be encoded into as:

      file://<host>/<absolute-path>
      

      Where, values of <host>> and <absolute-path> would be reported to the Ingest service by the client application qserv-replica-file-ingest.
      Later on, when the ingest service will be enchased to allow pulling table contributions over the HTTP protocol, other forms of the "source" names would be added. For example:

      http://<host>/<path>
      

      The corresponding REST service will be added to the Master Replication Controller to pull the bookkeeping records from the table and return them to a client.

      In a separate effort, the Qserv Dashboard Web application will be also extended to pull the bookkeeping records in order to display progress on the catalog ingests.

      Other actions:

      • update documentation on the extended REST services

        Attachments

          Issue Links

            Activity

            gapon Igor Gaponenko created issue -
            gapon Igor Gaponenko made changes -
            Field Original Value New Value
            Epic Link DM-25251 [ 435594 ]
            gapon Igor Gaponenko made changes -
            Description A goal of the effort is to add a persistent bookkeeping for table contributions made during catalog ingests. This would require creating a database table to store one record per each contribution. Each record would have:
            * the start time of the operation (when a request was received from a client)
            * the end time of the operation (when a request was accomplished or failed)
            * a completion status of the operation
            * (optional) an error code should the operation failed (ot enough storage, MySQL error, etc.)
            * an identifier of a transaction
            * the name of catalog
            * the base name of a table
            * (optionally) the chunk number (or {{NULL}})
            * (optionally) the {{overlap}} flag (or {{NULL}})
            * the number of rows ingested (if the operation succeeded)
            * (optionally, if known) a source of the contribution

            In the initial implementation (of the feature), the "source" name would be encoded into as:
            {code}
            file://&lt;host&gt;/&lt;absolute-path>
            {code}
            Where, values of {{<host>>}} and {{<absolute-path>}} would be reported to the Ingest service by the client application {{qserv-replica-file-ingest}}.

            A goal of the effort is to add a persistent bookkeeping for table contributions made during catalog ingests. This would require creating a database table to store one record per each contribution. Each record would have:
            * the start time of the operation (when a request was received from a client)
            * the end time of the operation (when a request was accomplished or failed)
            * a completion status of the operation
            * (optional) an error code should the operation failed (ot enough storage, MySQL error, etc.)
            * an identifier of a transaction
            * the name of catalog
            * the base name of a table
            * (optionally) the chunk number (or {{NULL}})
            * (optionally) the {{overlap}} flag (or {{NULL}})
            * the number of rows ingested (if the operation succeeded)
            * (optionally, if known) a source of the contribution

            In the initial implementation (of the feature), the "source" name would be encoded into as:
            {code}
            file://&lt;host&gt;/&lt;absolute-path>
            {code}
            Where, values of {{<host>>}} and {{<absolute-path>}} would be reported to the Ingest service by the client application {{qserv-replica-file-ingest}}.
            Later on, when the ingest service will be enchased to allow pulling table contributions over the HTTP protocol, other forms of the "source" names would be added. For example:
            {code}
            http://&lt;host&gt;/&lt;path>
            {code}

            The corresponding REST service will be added to the Master Replication Controller to pull the bookkeeping records from the table and return to a client.

            In a separate effort, the Qserv Dashboard Web application will be also extended to pull the bookkeeping records in order to display a progress on the catalog ingests.
            fritzm Fritz Mueller made changes -
            Sprint DB_F20_06 [ 1026 ] DB_F20_09 [ 1048 ]
            gapon Igor Gaponenko made changes -
            Description A goal of the effort is to add a persistent bookkeeping for table contributions made during catalog ingests. This would require creating a database table to store one record per each contribution. Each record would have:
            * the start time of the operation (when a request was received from a client)
            * the end time of the operation (when a request was accomplished or failed)
            * a completion status of the operation
            * (optional) an error code should the operation failed (ot enough storage, MySQL error, etc.)
            * an identifier of a transaction
            * the name of catalog
            * the base name of a table
            * (optionally) the chunk number (or {{NULL}})
            * (optionally) the {{overlap}} flag (or {{NULL}})
            * the number of rows ingested (if the operation succeeded)
            * (optionally, if known) a source of the contribution

            In the initial implementation (of the feature), the "source" name would be encoded into as:
            {code}
            file://&lt;host&gt;/&lt;absolute-path>
            {code}
            Where, values of {{<host>>}} and {{<absolute-path>}} would be reported to the Ingest service by the client application {{qserv-replica-file-ingest}}.
            Later on, when the ingest service will be enchased to allow pulling table contributions over the HTTP protocol, other forms of the "source" names would be added. For example:
            {code}
            http://&lt;host&gt;/&lt;path>
            {code}

            The corresponding REST service will be added to the Master Replication Controller to pull the bookkeeping records from the table and return to a client.

            In a separate effort, the Qserv Dashboard Web application will be also extended to pull the bookkeeping records in order to display a progress on the catalog ingests.
            A goal of the effort is to add persistent bookkeeping for table contributions made during catalog ingests. This would require creating a database table to store one record per each contribution. Each record would have:
            * the start time of the operation (when a request was received from a client)
            * the end time of the operation (when a request was accomplished or failed)
            * a completion status of the operation: {{IN-PROGRESS}}, {{SUCCESS}}, {{FAILED}}
            * (optional) an error code should the operation failed (MySQL error, etc.)
            * an identifier of a transaction (FK to the {{transaction}} table)
            * the name of the catalog (a part of the composite FK to column {{config_database_table.database}})
            * the base name of a table (a part of the composite FK to column {{config_database_table.table}})
            * the name of a worker
            * the chunk number (or {{NULL}} )
            * the {{overlap}} flag (or {{NULL}})
            * the number of rows ingested (if the operation succeeded)
            * the number of bytes ingested (if the operation succeeded)
            * (optionally, if known) a source of the contribution

            In the initial implementation (of the feature), the "source" name would be encoded into as:
            {code}
            file://&lt;host&gt;/&lt;absolute-path>
            {code}
            Where, values of {{<host>>}} and {{<absolute-path>}} would be reported to the Ingest service by the client application {{qserv-replica-file-ingest}}.
            Later on, when the ingest service will be enchased to allow pulling table contributions over the HTTP protocol, other forms of the "source" names would be added. For example:
            {code}
            http://&lt;host&gt;/&lt;path>
            {code}

            The corresponding REST service will be added to the Master Replication Controller to pull the bookkeeping records from the table and return them to a client.

            In a separate effort, the Qserv Dashboard Web application will be also extended to pull the bookkeeping records in order to display progress on the catalog ingests.
            gapon Igor Gaponenko made changes -
            Description A goal of the effort is to add persistent bookkeeping for table contributions made during catalog ingests. This would require creating a database table to store one record per each contribution. Each record would have:
            * the start time of the operation (when a request was received from a client)
            * the end time of the operation (when a request was accomplished or failed)
            * a completion status of the operation: {{IN-PROGRESS}}, {{SUCCESS}}, {{FAILED}}
            * (optional) an error code should the operation failed (MySQL error, etc.)
            * an identifier of a transaction (FK to the {{transaction}} table)
            * the name of the catalog (a part of the composite FK to column {{config_database_table.database}})
            * the base name of a table (a part of the composite FK to column {{config_database_table.table}})
            * the name of a worker
            * the chunk number (or {{NULL}} )
            * the {{overlap}} flag (or {{NULL}})
            * the number of rows ingested (if the operation succeeded)
            * the number of bytes ingested (if the operation succeeded)
            * (optionally, if known) a source of the contribution

            In the initial implementation (of the feature), the "source" name would be encoded into as:
            {code}
            file://&lt;host&gt;/&lt;absolute-path>
            {code}
            Where, values of {{<host>>}} and {{<absolute-path>}} would be reported to the Ingest service by the client application {{qserv-replica-file-ingest}}.
            Later on, when the ingest service will be enchased to allow pulling table contributions over the HTTP protocol, other forms of the "source" names would be added. For example:
            {code}
            http://&lt;host&gt;/&lt;path>
            {code}

            The corresponding REST service will be added to the Master Replication Controller to pull the bookkeeping records from the table and return them to a client.

            In a separate effort, the Qserv Dashboard Web application will be also extended to pull the bookkeeping records in order to display progress on the catalog ingests.
            A goal of the effort is to add persistent bookkeeping records for table contributions made during catalog ingests. This would require creating a database table to store one record per each contribution. Each record would have:
            * the start time of the operation (when a request was received from a client)
            * the end time of the operation (when a request was accomplished or failed)
            * a completion status of the operation: {{IN-PROGRESS}}, {{SUCCESS}}, {{FAILED}}
            * (optional) an error code should the operation failed (MySQL error, etc.)
            * an identifier of a transaction (FK to the {{transaction}} table)
            * the name of the catalog (a part of the composite FK to column {{config_database_table.database}})
            * the base name of a table (a part of the composite FK to column {{config_database_table.table}})
            * the name of a worker
            * the chunk number (or {{NULL}} )
            * the {{overlap}} flag (or {{NULL}})
            * the number of rows ingested (if the operation succeeded)
            * the number of bytes ingested (if the operation succeeded)
            * (optionally, if known) a source of the contribution

            In the initial implementation (of the feature), the "source" name would be encoded into as:
            {code}
            file://&lt;host&gt;/&lt;absolute-path>
            {code}
            Where, values of {{<host>>}} and {{<absolute-path>}} would be reported to the Ingest service by the client application {{qserv-replica-file-ingest}}.
            Later on, when the ingest service will be enchased to allow pulling table contributions over the HTTP protocol, other forms of the "source" names would be added. For example:
            {code}
            http://&lt;host&gt;/&lt;path>
            {code}

            The corresponding REST service will be added to the Master Replication Controller to pull the bookkeeping records from the table and return them to a client.

            In a separate effort, the Qserv Dashboard Web application will be also extended to pull the bookkeeping records in order to display progress on the catalog ingests.
            gapon Igor Gaponenko made changes -
            Description A goal of the effort is to add persistent bookkeeping records for table contributions made during catalog ingests. This would require creating a database table to store one record per each contribution. Each record would have:
            * the start time of the operation (when a request was received from a client)
            * the end time of the operation (when a request was accomplished or failed)
            * a completion status of the operation: {{IN-PROGRESS}}, {{SUCCESS}}, {{FAILED}}
            * (optional) an error code should the operation failed (MySQL error, etc.)
            * an identifier of a transaction (FK to the {{transaction}} table)
            * the name of the catalog (a part of the composite FK to column {{config_database_table.database}})
            * the base name of a table (a part of the composite FK to column {{config_database_table.table}})
            * the name of a worker
            * the chunk number (or {{NULL}} )
            * the {{overlap}} flag (or {{NULL}})
            * the number of rows ingested (if the operation succeeded)
            * the number of bytes ingested (if the operation succeeded)
            * (optionally, if known) a source of the contribution

            In the initial implementation (of the feature), the "source" name would be encoded into as:
            {code}
            file://&lt;host&gt;/&lt;absolute-path>
            {code}
            Where, values of {{<host>>}} and {{<absolute-path>}} would be reported to the Ingest service by the client application {{qserv-replica-file-ingest}}.
            Later on, when the ingest service will be enchased to allow pulling table contributions over the HTTP protocol, other forms of the "source" names would be added. For example:
            {code}
            http://&lt;host&gt;/&lt;path>
            {code}

            The corresponding REST service will be added to the Master Replication Controller to pull the bookkeeping records from the table and return them to a client.

            In a separate effort, the Qserv Dashboard Web application will be also extended to pull the bookkeeping records in order to display progress on the catalog ingests.
            A goal of the effort is to add persistent bookkeeping records for table contributions made during catalog ingests. This would require creating a database table to store one record per each contribution. Each record would have:
            * the start time of the operation (when a request was received from a client)
            * the end time of the operation (when a request was accomplished or failed)
            * a status of the operation: {{IN-PROGRESS}}, {{SUCCESS}}, {{FAILED}}
            * (optional) an error code should the operation failed (MySQL error, etc.)
            * an identifier of a transaction (FK to the {{transaction}} table)
            * the name of the catalog (a part of the composite FK to column {{config_database_table.database}})
            * the base name of a table (a part of the composite FK to column {{config_database_table.table}})
            * the name of a worker
            * the chunk number (or {{NULL}} )
            * the {{overlap}} flag (or {{NULL}})
            * the number of rows ingested (if the operation succeeded)
            * the number of bytes ingested (if the operation succeeded)
            * (optionally, if known) a source of the contribution

            In the initial implementation (of the feature), the "source" name would be encoded into as:
            {code}
            file://&lt;host&gt;/&lt;absolute-path>
            {code}
            Where, values of {{<host>>}} and {{<absolute-path>}} would be reported to the Ingest service by the client application {{qserv-replica-file-ingest}}.
            Later on, when the ingest service will be enchased to allow pulling table contributions over the HTTP protocol, other forms of the "source" names would be added. For example:
            {code}
            http://&lt;host&gt;/&lt;path>
            {code}

            The corresponding REST service will be added to the Master Replication Controller to pull the bookkeeping records from the table and return them to a client.

            In a separate effort, the Qserv Dashboard Web application will be also extended to pull the bookkeeping records in order to display progress on the catalog ingests.
            fritzm Fritz Mueller made changes -
            Epic Link DM-25251 [ 435594 ] DM-27790 [ 442306 ]
            fritzm Fritz Mueller made changes -
            Sprint DB_F20_09 [ 1048 ] DB_F20_09, DB_S21_12 [ 1048, 1065 ]
            gapon Igor Gaponenko made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            gapon Igor Gaponenko made changes -
            Story Points 2 6
            gapon Igor Gaponenko made changes -
            Story Points 6 8
            gapon Igor Gaponenko made changes -
            Link This issue blocks DM-19630 [ DM-19630 ]
            gapon Igor Gaponenko made changes -
            Reviewers John Gates, Nate Pease [ jgates, npease ]
            gapon Igor Gaponenko made changes -
            Status In Progress [ 3 ] In Review [ 10004 ]
            gapon Igor Gaponenko made changes -
            Reviewers John Gates, Nate Pease [ jgates, npease ] Nate Pease [ npease ]
            gapon Igor Gaponenko made changes -
            Link This issue blocks DM-28603 [ DM-28603 ]
            npease Nate Pease made changes -
            Status In Review [ 10004 ] Reviewed [ 10101 ]
            gapon Igor Gaponenko made changes -
            Description A goal of the effort is to add persistent bookkeeping records for table contributions made during catalog ingests. This would require creating a database table to store one record per each contribution. Each record would have:
            * the start time of the operation (when a request was received from a client)
            * the end time of the operation (when a request was accomplished or failed)
            * a status of the operation: {{IN-PROGRESS}}, {{SUCCESS}}, {{FAILED}}
            * (optional) an error code should the operation failed (MySQL error, etc.)
            * an identifier of a transaction (FK to the {{transaction}} table)
            * the name of the catalog (a part of the composite FK to column {{config_database_table.database}})
            * the base name of a table (a part of the composite FK to column {{config_database_table.table}})
            * the name of a worker
            * the chunk number (or {{NULL}} )
            * the {{overlap}} flag (or {{NULL}})
            * the number of rows ingested (if the operation succeeded)
            * the number of bytes ingested (if the operation succeeded)
            * (optionally, if known) a source of the contribution

            In the initial implementation (of the feature), the "source" name would be encoded into as:
            {code}
            file://&lt;host&gt;/&lt;absolute-path>
            {code}
            Where, values of {{<host>>}} and {{<absolute-path>}} would be reported to the Ingest service by the client application {{qserv-replica-file-ingest}}.
            Later on, when the ingest service will be enchased to allow pulling table contributions over the HTTP protocol, other forms of the "source" names would be added. For example:
            {code}
            http://&lt;host&gt;/&lt;path>
            {code}

            The corresponding REST service will be added to the Master Replication Controller to pull the bookkeeping records from the table and return them to a client.

            In a separate effort, the Qserv Dashboard Web application will be also extended to pull the bookkeeping records in order to display progress on the catalog ingests.
            A goal of the effort is to add persistent bookkeeping records for table contributions made during catalog ingests. This would require creating a database table to store one record per each contribution. Each record would have:
            * the start time of the operation (when a request was received from a client)
            * the end time of the operation (when a request was accomplished or failed)
            * a status of the operation: {{IN-PROGRESS}}, {{SUCCESS}}, {{FAILED}}
            * (optional) an error code should the operation failed (MySQL error, etc.)
            * an identifier of a transaction (FK to the {{transaction}} table)
            * the name of the catalog (a part of the composite FK to column {{config_database_table.database}})
            * the base name of a table (a part of the composite FK to column {{config_database_table.table}})
            * the name of a worker
            * the chunk number (or {{NULL}} )
            * the {{overlap}} flag (or {{NULL}})
            * the number of rows ingested (if the operation succeeded)
            * the number of bytes ingested (if the operation succeeded)
            * (optionally, if known) a source of the contribution

            In the initial implementation (of the feature), the "source" name would be encoded into as:
            {code}
            file://&lt;host&gt;/&lt;absolute-path>
            {code}
            Where, values of {{<host>>}} and {{<absolute-path>}} would be reported to the Ingest service by the client application {{qserv-replica-file-ingest}}.
            Later on, when the ingest service will be enchased to allow pulling table contributions over the HTTP protocol, other forms of the "source" names would be added. For example:
            {code}
            http://&lt;host&gt;/&lt;path>
            {code}

            The corresponding REST service will be added to the Master Replication Controller to pull the bookkeeping records from the table and return them to a client.

            In a separate effort, the Qserv Dashboard Web application will be also extended to pull the bookkeeping records in order to display progress on the catalog ingests.

            Other actions:
            * update documentation on the extended REST services
            gapon Igor Gaponenko made changes -
            Resolution Done [ 10000 ]
            Status Reviewed [ 10101 ] Done [ 10002 ]
            gapon Igor Gaponenko made changes -
            Link This issue is triggering DM-30095 [ DM-30095 ]

              People

              Assignee:
              gapon Igor Gaponenko
              Reporter:
              gapon Igor Gaponenko
              Reviewers:
              Nate Pease
              Watchers:
              Fritz Mueller, Igor Gaponenko, John Gates, Nate Pease
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  CI Builds

                  No builds found.