Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-2760

Data loader fails to decompress compressed data

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: Qserv
    • Labels:

      Description

      In some condition data loader decides to not uncompress data files before loading them directly into database table. No errors are generated in this case, but the result is funny.

        Attachments

          Activity

          Hide
          salnikov Andy Salnikov added a comment -

          This command loads data into a table just fine:

          qserv-data-loader.py -v -v --config=/usr/local/home/salnikov/testdata-repo-master/datasets/case01/data/common.cfg --user=root --password=changeme --socket=/usr/local/home/salnikov/qserv-run/2015_05/var/lib/mysql/mysql.sock --delete-tables --chunks-dir=/usr/local/home/salnikov/qserv-run/2015_05/tmp/qserv_data_loader/Object --no-css --skip-partition --one-table qservTest_case01_mysql Object /usr/local/home/salnikov/testdata-repo-master/datasets/case01/data/Object.schema /usr/local/home/salnikov/testdata-repo-master/datasets/case01/data/Object.tsv.gz

          But this loads the data without uncompressing them:

          qserv-data-loader.py -v -v --config=/usr/local/home/salnikov/testdata-repo-master/datasets/case01/data/common.cfg --user=root --password=changeme --socket=/usr/local/home/salnikov/qserv-run/2015_05/var/lib/mysql/mysql.sock --delete-tables --chunks-dir=/usr/local/home/salnikov/qserv-run/2015_05/tmp/qserv_data_loader/Object --no-css --skip-partition --one-table --config=/usr/local/home/salnikov/testdata-repo-master/datasets/case01/data/Object.cfg qservTest_case01_mysql Object /usr/local/home/salnikov/testdata-repo-master/datasets/case01/data/Object.schema /usr/local/home/salnikov/testdata-repo-master/datasets/case01/data/Object.tsv.gz

          The difference is extra --config=/usr/local/home/salnikov/testdata-repo-master/datasets/case01/data/Object.cfg option.

          Show
          salnikov Andy Salnikov added a comment - This command loads data into a table just fine: qserv-data-loader.py -v -v --config=/usr/local/home/salnikov/testdata-repo-master/datasets/case01/data/common.cfg --user=root --password=changeme --socket=/usr/local/home/salnikov/qserv-run/2015_05/var/lib/mysql/mysql.sock --delete-tables --chunks-dir=/usr/local/home/salnikov/qserv-run/2015_05/tmp/qserv_data_loader/Object --no-css --skip-partition --one-table qservTest_case01_mysql Object /usr/local/home/salnikov/testdata-repo-master/datasets/case01/data/Object.schema /usr/local/home/salnikov/testdata-repo-master/datasets/case01/data/Object.tsv.gz But this loads the data without uncompressing them: qserv-data-loader.py -v -v --config=/usr/local/home/salnikov/testdata-repo-master/datasets/case01/data/common.cfg --user=root --password=changeme --socket=/usr/local/home/salnikov/qserv-run/2015_05/var/lib/mysql/mysql.sock --delete-tables --chunks-dir=/usr/local/home/salnikov/qserv-run/2015_05/tmp/qserv_data_loader/Object --no-css --skip-partition --one-table --config=/usr/local/home/salnikov/testdata-repo-master/datasets/case01/data/Object.cfg qservTest_case01_mysql Object /usr/local/home/salnikov/testdata-repo-master/datasets/case01/data/Object.schema /usr/local/home/salnikov/testdata-repo-master/datasets/case01/data/Object.tsv.gz The difference is extra --config=/usr/local/home/salnikov/testdata-repo-master/datasets/case01/data/Object.cfg option.
          Hide
          salnikov Andy Salnikov added a comment -

          Logical mistake when determining when to uncompress the file. There is a trivial fix, I'm going to self-review it and merge to master.

          Show
          salnikov Andy Salnikov added a comment - Logical mistake when determining when to uncompress the file. There is a trivial fix, I'm going to self-review it and merge to master.

            People

            Assignee:
            salnikov Andy Salnikov
            Reporter:
            salnikov Andy Salnikov
            Watchers:
            Andy Salnikov
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                CI Builds

                No builds found.