Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-12187

Fixes during end-to-end testing with validate_drp and post-qa

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Issues found during the end-to-end test of the squash-deployment

        Attachments

          Issue Links

            Activity

            Hide
            afausti Angelo Fausti added a comment - - edited

            Summary of the changes:

            • From the squash-api nginx log:

            413 Client Error: Request Entity Too Large for url: https://squash-api-demo.lsst.codes/jobs/
            

            we increased the max allowed size of the http request body to 1G in the squash-api nginx configuration

            client_max_body_size 1024m;
            

            • From the mariadb log:

            2017-10-05 23:55:41 14 [Warning] Aborted connection 14 to db: 'qadb' user: 'root' host: '10.44.2.49' (Got an error reading communication packets)
            

            we had to increase the size of the innodb log file in the mariadb config

            innodb_log_file_size    = 3G
            

            • django 1.11.3 does not play well with django-json-field==0.5.7, this package is used to deserialize the JSONField columns in the database. Added more tests to the squash-api and decided to roll back to django 1.8.4 used in the old deployment. There are improvements to the squash-api to be described and implemented in the next cycle.
            • MariaDB 10.3 used in the new deployment does not create datetime records with microseconds by default

            MariaDB [qadb]> select NOW();
            +---------------------+
            | NOW()               |
            +---------------------+
            | 2017-10-10 01:46:35 |
            +---------------------+
            1 row in set (0.000 sec)
            
            

            then the creation date in the api_job table when we do the POST in the /jobs endpoint had a different format compared to the dates restored from the old production database.

            The solution was to modify the DateTimeField field type from datetime(6) to datetime after restoring the old production db.

            MariaDB [qadb]> alter table api_job modify date datetime;
            
            

            this is something that can be controlled in the django if we subclass the DateTimeField to handle the microsecond precision explicitly, see discussion here:

            https://groups.google.com/forum/#!topic/django-developers/kh5Iia0uG6E

            Show
            afausti Angelo Fausti added a comment - - edited Summary of the changes: From the squash-api nginx log: 413 Client Error: Request Entity Too Large for url: https: //squash-api-demo.lsst.codes/jobs/ we increased the max allowed size of the http request body to 1G in the squash-api nginx configuration client_max_body_size 1024m; From the mariadb log: 2017 - 10 - 05 23 : 55 : 41 14 [Warning] Aborted connection 14 to db: 'qadb' user: 'root' host: '10.44.2.49' (Got an error reading communication packets) we had to increase the size of the innodb log file in the mariadb config innodb_log_file_size = 3G django 1.11.3 does not play well with django-json-field==0.5.7 , this package is used to deserialize the JSONField columns in the database. Added more tests to the squash-api and decided to roll back to django 1.8.4 used in the old deployment. There are improvements to the squash-api to be described and implemented in the next cycle. MariaDB 10.3 used in the new deployment does not create datetime records with microseconds by default MariaDB [qadb]> select NOW(); +---------------------+ | NOW() | +---------------------+ | 2017 - 10 - 10 01 : 46 : 35 | +---------------------+ 1 row in set ( 0.000 sec) then the creation date in the api_job table when we do the POST in the /jobs endpoint had a different format compared to the dates restored from the old production database. The solution was to modify the DateTimeField field type from datetime(6) to datetime after restoring the old production db. MariaDB [qadb]> alter table api_job modify date datetime; this is something that can be controlled in the django if we subclass the DateTimeField to handle the microsecond precision explicitly, see discussion here: https://groups.google.com/forum/#!topic/django-developers/kh5Iia0uG6E
            Hide
            afausti Angelo Fausti added a comment - - edited
            Show
            afausti Angelo Fausti added a comment - - edited New squash deployment at https://squash-dash-demo.lsst.codes/ End-to-end tests are passing now, using post-qa 1.3.1 https://ci.lsst.codes/job/sqre/job/validate_drp-test/ Jenkins job configuration is here https://github.com/lsst-sqre/jenkins-dm-jobs/blob/master/jobs/validate_drp_test.groovy
            Show
            afausti Angelo Fausti added a comment - Changes applied to the following repos: https://github.com/lsst-sqre/squash-db/pull/5 https://github.com/lsst-sqre/squash-api/pull/5 https://github.com/lsst-sqre/squash-bokeh/pull/5 https://github.com/lsst-sqre/squash-deployment/pull/7
            Hide
            afausti Angelo Fausti added a comment - - edited

            Tested in `squash-demo` deployment, will move on and apply these fixes to `squash-prod` deployment.

            Show
            afausti Angelo Fausti added a comment - - edited Tested in `squash-demo` deployment, will move on and apply these fixes to `squash-prod` deployment.

              People

              Assignee:
              afausti Angelo Fausti
              Reporter:
              afausti Angelo Fausti
              Reviewers:
              Joshua Hoblitt
              Watchers:
              Angelo Fausti, Joshua Hoblitt
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins Builds

                  No builds found.