Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-12187

Fixes during end-to-end testing with validate_drp and post-qa

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Issues found during the end-to-end test of the squash-deployment

        Attachments

          Issue Links

            Activity

            No builds found.
            afausti Angelo Fausti created issue -
            afausti Angelo Fausti made changes -
            Field Original Value New Value
            Epic Link DM-10742 [ 32696 ]
            afausti Angelo Fausti made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            afausti Angelo Fausti made changes -
            Description Issues found during the end-to-end test of the {{squash-deployment}}
            Hide
            afausti Angelo Fausti added a comment - - edited

            Summary of the changes:

            • From the squash-api nginx log:

            413 Client Error: Request Entity Too Large for url: https://squash-api-demo.lsst.codes/jobs/
            

            we increased the max allowed size of the http request body to 1G in the squash-api nginx configuration

            client_max_body_size 1024m;
            

            • From the mariadb log:

            2017-10-05 23:55:41 14 [Warning] Aborted connection 14 to db: 'qadb' user: 'root' host: '10.44.2.49' (Got an error reading communication packets)
            

            we had to increase the size of the innodb log file in the mariadb config

            innodb_log_file_size    = 3G
            

            • django 1.11.3 does not play well with django-json-field==0.5.7, this package is used to deserialize the JSONField columns in the database. Added more tests to the squash-api and decided to roll back to django 1.8.4 used in the old deployment. There are improvements to the squash-api to be described and implemented in the next cycle.
            • MariaDB 10.3 used in the new deployment does not create datetime records with microseconds by default

            MariaDB [qadb]> select NOW();
            +---------------------+
            | NOW()               |
            +---------------------+
            | 2017-10-10 01:46:35 |
            +---------------------+
            1 row in set (0.000 sec)
            
            

            then the creation date in the api_job table when we do the POST in the /jobs endpoint had a different format compared to the dates restored from the old production database.

            The solution was to modify the DateTimeField field type from datetime(6) to datetime after restoring the old production db.

            MariaDB [qadb]> alter table api_job modify date datetime;
            
            

            this is something that can be controlled in the django if we subclass the DateTimeField to handle the microsecond precision explicitly, see discussion here:

            https://groups.google.com/forum/#!topic/django-developers/kh5Iia0uG6E

            Show
            afausti Angelo Fausti added a comment - - edited Summary of the changes: From the squash-api nginx log: 413 Client Error: Request Entity Too Large for url: https: //squash-api-demo.lsst.codes/jobs/ we increased the max allowed size of the http request body to 1G in the squash-api nginx configuration client_max_body_size 1024m; From the mariadb log: 2017 - 10 - 05 23 : 55 : 41 14 [Warning] Aborted connection 14 to db: 'qadb' user: 'root' host: '10.44.2.49' (Got an error reading communication packets) we had to increase the size of the innodb log file in the mariadb config innodb_log_file_size = 3G django 1.11.3 does not play well with django-json-field==0.5.7 , this package is used to deserialize the JSONField columns in the database. Added more tests to the squash-api and decided to roll back to django 1.8.4 used in the old deployment. There are improvements to the squash-api to be described and implemented in the next cycle. MariaDB 10.3 used in the new deployment does not create datetime records with microseconds by default MariaDB [qadb]> select NOW(); +---------------------+ | NOW() | +---------------------+ | 2017 - 10 - 10 01 : 46 : 35 | +---------------------+ 1 row in set ( 0.000 sec) then the creation date in the api_job table when we do the POST in the /jobs endpoint had a different format compared to the dates restored from the old production database. The solution was to modify the DateTimeField field type from datetime(6) to datetime after restoring the old production db. MariaDB [qadb]> alter table api_job modify date datetime; this is something that can be controlled in the django if we subclass the DateTimeField to handle the microsecond precision explicitly, see discussion here: https://groups.google.com/forum/#!topic/django-developers/kh5Iia0uG6E
            Hide
            afausti Angelo Fausti added a comment - - edited
            Show
            afausti Angelo Fausti added a comment - - edited New squash deployment at https://squash-dash-demo.lsst.codes/ End-to-end tests are passing now, using post-qa 1.3.1 https://ci.lsst.codes/job/sqre/job/validate_drp-test/ Jenkins job configuration is here https://github.com/lsst-sqre/jenkins-dm-jobs/blob/master/jobs/validate_drp_test.groovy
            afausti Angelo Fausti made changes -
            Story Points 2.8
            afausti Angelo Fausti made changes -
            Summary Fixes in squash after testing new deployment with validate_drp and post-qa Fixes during end-to-end testing with validate_drp and post-qa
            Show
            afausti Angelo Fausti added a comment - Changes applied to the following repos: https://github.com/lsst-sqre/squash-db/pull/5 https://github.com/lsst-sqre/squash-api/pull/5 https://github.com/lsst-sqre/squash-bokeh/pull/5 https://github.com/lsst-sqre/squash-deployment/pull/7
            afausti Angelo Fausti made changes -
            Reviewers Joshua Hoblitt [ jhoblitt ]
            Status In Progress [ 3 ] In Review [ 10004 ]
            Hide
            afausti Angelo Fausti added a comment - - edited

            Tested in `squash-demo` deployment, will move on and apply these fixes to `squash-prod` deployment.

            Show
            afausti Angelo Fausti added a comment - - edited Tested in `squash-demo` deployment, will move on and apply these fixes to `squash-prod` deployment.
            afausti Angelo Fausti made changes -
            Resolution Done [ 10000 ]
            Status In Review [ 10004 ] Done [ 10002 ]
            afausti Angelo Fausti made changes -
            Link This issue is triggering DM-12192 [ DM-12192 ]

              People

              Assignee:
              afausti Angelo Fausti
              Reporter:
              afausti Angelo Fausti
              Reviewers:
              Joshua Hoblitt
              Watchers:
              Angelo Fausti, Joshua Hoblitt
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.