Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-8001

Improvements to the database loading procedure to allow parallel loading of chunks

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: Qserv
    • Labels:

      Description

      The rationale

      A possible workflow for loading a large catalog into Qserv may require various forms of parallelism both at the data preparation stage and when performing the actual loading. This opens a possibility that the actual data loaded stage may be operating on fragments of the same chunk spread between multiple input files. The current implementation of the database loading procedure is not safe to operate in the input data structured in this way. Neither it would work for subsequent loading fragments of the same chunk. There are two failure modes which would prevent the current procedure from succeeding:

      • chunk-to-worker node mapping: each instance of the loading script will try to follow its own round-robin allocation sequence of worker nodes for chunks as they (the chunks) are discovered in an input folder. Since no coordination has been employed so far between multiple instances of the loading script two instances may try to allocate different worker nodes for the same chunk. This would be a problem when an input data contains chunks split into fragments.
      • chunk creation: the current implementation of the loading script is based on an assumption that the loading is always made on a fresh (empty) instance of Qserv. As a result of this the script won't check if a chunk structure already exists on the corresponding worker node. This may cause the script to fail when attempting to add more data to an existing chunk.

      The proposed solution

      A solution to the chunk-to-worker node mapping problem would be to pre-populate the CSS database with the mapping before loading any chunks into Qserv.

      A solution to the second problem (of the chunk creation) is to reinforce the implementation of the loading procedure to detect an existence of a chunk on a worker node and not to fail if the one already exists (was created earlier by some other instance of the loader run either in parallel or before this one).

        Attachments

          Issue Links

            Activity

            Hide
            gapon Igor Gaponenko added a comment -

            The progress of the development

            Created branch tickets/DM-8001 based on master of package qserv.

            Extended CSS tools in qserv-admin.py

            The modified version of the the application supports 3 additional commands:

            CREATE CHUNK <db>.<table> <chunk> <worker>
            DELETE CHUNK <db>.<table> <chunk>
            SHOW CHUNKS <db>.<table>
            

            This also required extending the corresponding C++ module to support the chunk deleton operation.

            Show
            gapon Igor Gaponenko added a comment - The progress of the development Created branch tickets/ DM-8001 based on master of package qserv . Extended CSS tools in qserv-admin.py The modified version of the the application supports 3 additional commands: CREATE CHUNK <db>.< table > <chunk> <worker> DELETE CHUNK <db>.< table > <chunk> SHOW CHUNKS <db>.< table > This also required extending the corresponding C++ module to support the chunk deleton operation.
            Hide
            gapon Igor Gaponenko added a comment - - edited

            Fixing a bug in qserv-admin.py

            Ran into the following bug when trying to use the tool to explore the contents of the CSS database:

            qserv-admin.py 'DUMP EVERYTHING;'
            ERROR:  KvInterfaceImplMySql::dumpKV - value contains newline
            

            Further investigation has pointed to the multi-line keys with table schema. Examples:

            /DBS/sdss_stripe82_00/TABLES/Science_Ccd_Exposure/schema
            ...
            

            The problem was found and fixed in the following file witin qserv package:

            core/modules/css/KvInterfaceImplMySql.cc
            

            Show
            gapon Igor Gaponenko added a comment - - edited Fixing a bug in qserv-admin.py Ran into the following bug when trying to use the tool to explore the contents of the CSS database: qserv-admin.py 'DUMP EVERYTHING;' ERROR: KvInterfaceImplMySql::dumpKV - value contains newline Further investigation has pointed to the multi-line keys with table schema. Examples: /DBS/sdss_stripe82_00/TABLES/Science_Ccd_Exposure/schema ... The problem was found and fixed in the following file witin qserv package: core/modules/css/KvInterfaceImplMySql.cc
            Hide
            gapon Igor Gaponenko added a comment -

            Improved error handling when creating tables

            Capture HTTP error code 409 (returned by wmgr when a table already exists) and report the one within the logger's INFO category. This applies to both regular and partitioned tables.

            The new version of the code allows partial loading of large tables.
            NOTE: special care must be taken when an existing table needs to be cleaned up.

            Relaxed logic for creating the "Secondary" index table

            Do not attempt destroying that table before creating the one. Always create the table using this SQL statement:

            CREATE TABLE IF NOT EXISTS...
            

            The new logic allows multiple loaders to extend the index w/o running into a race condition.

            Show
            gapon Igor Gaponenko added a comment - Improved error handling when creating tables Capture HTTP error code 409 (returned by wmgr when a table already exists) and report the one within the logger's INFO category. This applies to both regular and partitioned tables. The new version of the code allows partial loading of large tables. NOTE : special care must be taken when an existing table needs to be cleaned up. Relaxed logic for creating the "Secondary" index table Do not attempt destroying that table before creating the one. Always create the table using this SQL statement: CREATE TABLE IF NOT EXISTS... The new logic allows multiple loaders to extend the index w/o running into a race condition.
            Hide
            gapon Igor Gaponenko added a comment -

            Andy,

            please review my modifications! I would like to close this ticket before Thursday (end of the current Sprint).

            Thanks!
            Igor

            Show
            gapon Igor Gaponenko added a comment - Andy, please review my modifications! I would like to close this ticket before Thursday (end of the current Sprint). Thanks! Igor
            Hide
            salnikov Andy Salnikov added a comment -

            Looks good, I left few comments on PR.

            Show
            salnikov Andy Salnikov added a comment - Looks good, I left few comments on PR.

              People

              Assignee:
              gapon Igor Gaponenko
              Reporter:
              gapon Igor Gaponenko
              Reviewers:
              Andy Salnikov
              Watchers:
              Andy Salnikov, Fritz Mueller, Igor Gaponenko
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: