Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-26101

Add bookkeeping for table contributions requested during catalog ingests

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: Qserv
    • Labels:
      None

      Description

      A goal of the effort is to add persistent bookkeeping records for table contributions made during catalog ingests. This would require creating a database table to store one record per each contribution. Each record would have:

      • the start time of the operation (when a request was received from a client)
      • the end time of the operation (when a request was accomplished or failed)
      • a status of the operation: IN-PROGRESS, SUCCESS, FAILED
      • (optional) an error code should the operation failed (MySQL error, etc.)
      • an identifier of a transaction (FK to the transaction table)
      • the name of the catalog (a part of the composite FK to column config_database_table.database)
      • the base name of a table (a part of the composite FK to column config_database_table.table)
      • the name of a worker
      • the chunk number (or NULL )
      • the overlap flag (or NULL)
      • the number of rows ingested (if the operation succeeded)
      • the number of bytes ingested (if the operation succeeded)
      • (optionally, if known) a source of the contribution

      In the initial implementation (of the feature), the "source" name would be encoded into as:

      file://<host>/<absolute-path>
      

      Where, values of <host>> and <absolute-path> would be reported to the Ingest service by the client application qserv-replica-file-ingest.
      Later on, when the ingest service will be enchased to allow pulling table contributions over the HTTP protocol, other forms of the "source" names would be added. For example:

      http://<host>/<path>
      

      The corresponding REST service will be added to the Master Replication Controller to pull the bookkeeping records from the table and return them to a client.

      In a separate effort, the Qserv Dashboard Web application will be also extended to pull the bookkeeping records in order to display progress on the catalog ingests.

      Other actions:

      • update documentation on the extended REST services

        Attachments

          Issue Links

            Activity

            Hide
            npease Nate Pease added a comment -

            looks good

            Show
            npease Nate Pease added a comment - looks good

              People

              Assignee:
              gapon Igor Gaponenko
              Reporter:
              gapon Igor Gaponenko
              Reviewers:
              Nate Pease
              Watchers:
              Fritz Mueller, Igor Gaponenko, John Gates, Nate Pease
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: