Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-26532

Optimize qserv-ingest for DC2 and add support for GKE

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: Qserv
    • Labels:
      None
    • Story Points:
      10
    • Sprint:
      DB_F20_09, DB_S21_12
    • Team:
      Data Access and Database
    • Urgent?:
      No

      Description

      • Integrate ReplicationService new version (with avoid moving the input file twice over the network)
      • Optimization: 2nd index generation at a time: https://confluence.lsstcorp.org/pages/viewpage.action?pageId=133333850#UserguidefortheQservIngestsystem(APIversion2)-Secondaryindexmanagement
      • ingest of batch files, UNIQUE keys
      • Monitoring and observability
      • Semi automated error management and related documentation
      • Ingest of overlap files for non-director table
      • Test of replication service support for k8s dynamic DNS
      • Fix query error:

        mysql --protocol=tcp -hlocalhost -uqsmaster -P4040 \
           -e 'SELECT coord_ra, coord_dec FROM dc2_run2_1i_dr1b.dpdd_ref WHERE qserv_areaspec_box(62.94,-40,63.65,-39.30)'
        ERROR 4120 (Proxy) at line 1: Unable to return query results:
        [1470] Error(s) in result for chunk #1234567890: [1054] primeResult error Unknown column 'dc2_run2_1i_dr1b.dpdd_ref.' in 'where clause'

      • use alpine as base image
      • solve deadlock error on chunk queue deletion

        Attachments

          Issue Links

            Activity

            Hide
            gapon Igor Gaponenko added a comment -

            I'm not an expert in GoLang. However, what I've seen looks good to me.

            Show
            gapon Igor Gaponenko added a comment - I'm not an expert in GoLang. However, what I've seen looks good to me.

              People

              Assignee:
              jammes Fabrice Jammes
              Reporter:
              jammes Fabrice Jammes
              Reviewers:
              Igor Gaponenko
              Watchers:
              Dominique Boutigny, Fabrice Jammes, Fritz Mueller, Hsin-Fang Chiang, Igor Gaponenko, Sabine Elles
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: