Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-28626

Allow un-publishing catalogs in the Qserv Replication/Ingest system

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: To Do
    • Resolution: Unresolved
    • Fix Version/s: None
    • Component/s: Qserv
    • Labels:
      None

      Description

      Goals

      The amount of effort needed to implement this feature request is unpredictable at this stage. A preliminary study is required to come up with the right number here.

      The first step would be to:

      • investigate the possibility (and consequences) of putting a published catalog back into the "un-published" state so that more products could be added to the catalog.
      • study an option that would also allow modifying existing products (for example: adding more rows to the tables, adding more chunks). Take into account the effect of dealing with multiple replicas (if any existed) of the partitioned tables distributed across workers.
        • NOTE: this may be tricky to implement since MySQL partitions get eliminated from tables by the catalog publishing algorithm. One possibility would be to enable MySQL partitions in an existing monolithic table by putting all existing data into a special partition corresponding to the pseudo-transaction id=0?
      • plan for improvements to the catalog publishing operation to avoid re-publishing consolidated no MySQL partitions) tables that were not modified by the additional ingest into the catalog.

      For the end-users of the Qserv Ingest system, the new feature will be available via a dedicated REST service to be added to the Master Replication Controller

        Attachments

          Activity

          There are no comments yet on this issue.

            People

            Assignee:
            gapon Igor Gaponenko
            Reporter:
            gapon Igor Gaponenko
            Watchers:
            Fabrice Jammes, Fritz Mueller, Igor Gaponenko, Nate Pease
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Dates

              Created:
              Updated: