Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-27629

Stop creating MySQL partitions in the DUMMY chunk tables

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: Qserv
    • Labels:
      None

      Description

      A problem has been found with the current implementation of the worker ingest service. When processing contributions into a given chunk the service adds a MySQL partition corresponding to the specified "super-transaction" for all relevant tables. This operation also includes the so-called "DUMMY" chunk number 1234567890. Unfortunately, this algorithm leaves the "DUMMY" chunk tables fragmented into as many MySQL partitions as many "super-transactions" were started by an ingest workflow during catalog ingest. This turns the "DUMMY" tables into hot spots affecting many stages of the catalog ingest workflow.

      Further analysis of this situation has revealed that making MySQL partitions for all bit the default transaction (transaction_id=0) doesn't make sense since "DUMMY" chunk tables are not supposed to store any data in Qserv.

      This effort is meant to optimize existing code and exclude the "DUMMY" tables from the MySQL partitions creation for explicitly started transactions (other than the default one transaction_id=0).

        Attachments

          Activity

          Show
          gapon Igor Gaponenko added a comment - PR: https://github.com/lsst/qserv/pull/580
          Hide
          jgates John Gates added a comment -

          Looks good.

          Show
          jgates John Gates added a comment - Looks good.

            People

            Assignee:
            gapon Igor Gaponenko
            Reporter:
            gapon Igor Gaponenko
            Reviewers:
            John Gates
            Watchers:
            Fabrice Jammes, Fritz Mueller, Igor Gaponenko, John Gates, Nate Pease
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved: