Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-9354

qadb missing migration

    Details

      Description

      Per DM-8794, it was discovered that squash from from the production branch seems to be missing a nessicary qadb schema change.

      Feb 09 10:15:09 jenkins-squash scl[11049]: 2017-02-09 10:15:09,398 WebSocket connection closed: code=1001, reason=None
      Feb 09 10:15:09 jenkins-squash uwsgi[11029]: [pid: 11037|app: 0|req: 3/13] 127.0.0.1 () {48 vars in 958 bytes} [Thu Feb  9 18:15:09 2017] GET /dashboard => generated 2646 bytes in 4 msecs (HTTP/1.1 200) 2 headers in 88 bytes (1 switches on core 0)
      Feb 09 10:15:09 jenkins-squash oauth2_proxy[10410]: 67.212.196.101 - josh@hoblitt.com [09/Feb/2017:10:15:09 -0800] jhoblitt-larry-squash.lsst.codes GET 127.0.0.1:9090 "/dashboard" HTTP/1.0 "Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:51.0) Gecko/20100101 Firefox/51.0" 200 2646 0.005
      Feb 09 10:15:09 jenkins-squash scl[11049]: 2017-02-09 10:15:09,545 Error running application handler <bokeh.application.handlers.script.ScriptHandler object at 0x7f34f9b1f048>: Expecting value: line 1 column 1 (char 0)
      Feb 09 10:15:09 jenkins-squash scl[11049]: File "decoder.py", line 357, in raw_decode:
      Feb 09 10:15:09 jenkins-squash scl[11049]: raise JSONDecodeError("Expecting value", s, err.value) from None Traceback (most recent call last):
      Feb 09 10:15:09 jenkins-squash scl[11049]: File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/bokeh/application/handlers/code_runner.py", line 83, in run
      Feb 09 10:15:09 jenkins-squash scl[11049]: exec(self._code, module.__dict__)
      Feb 09 10:15:09 jenkins-squash scl[11049]: File "/opt/apps/qa-dashboard/squash/dashboard/viz/metrics.py", line 346, in <module>
      Feb 09 10:15:09 jenkins-squash scl[11049]: curdoc().add_root(Metrics().layout)
      Feb 09 10:15:09 jenkins-squash scl[11049]: File "/opt/apps/qa-dashboard/squash/dashboard/viz/metrics.py", line 32, in __init__
      Feb 09 10:15:09 jenkins-squash scl[11049]: self.compose_layout()
      Feb 09 10:15:09 jenkins-squash scl[11049]: File "/opt/apps/qa-dashboard/squash/dashboard/viz/metrics.py", line 74, in compose_layout
      Feb 09 10:15:09 jenkins-squash scl[11049]: self.selected_window)
      Feb 09 10:15:09 jenkins-squash scl[11049]: File "/opt/apps/qa-dashboard/squash/dashboard/viz/service.py", line 131, in get_meas_by_dataset_and_metric
      Feb 09 10:15:09 jenkins-squash scl[11049]: 'metric': selected_metric}).json()
      Feb 09 10:15:09 jenkins-squash scl[11049]: File "/opt/apps/qa-dashboard/venv/lib/python3.5/site-packages/requests/models.py", line 808, in json
      Feb 09 10:15:09 jenkins-squash scl[11049]: return complexjson.loads(self.text, **kwargs)
      Feb 09 10:15:09 jenkins-squash scl[11049]: File "/opt/rh/rh-python35/root/usr/lib64/python3.5/json/__init__.py", line 319, in loads
      Feb 09 10:15:09 jenkins-squash scl[11049]: return _default_decoder.decode(s)
      Feb 09 10:15:09 jenkins-squash scl[11049]: File "/opt/rh/rh-python35/root/usr/lib64/python3.5/json/decoder.py", line 339, in decode
      Feb 09 10:15:09 jenkins-squash scl[11049]: obj, end = self.raw_decode(s, idx=_w(s, 0).end())
      Feb 09 10:15:09 jenkins-squash scl[11049]: File "/opt/rh/rh-python35/root/usr/lib64/python3.5/json/decoder.py", line 357, in raw_decode
      Feb 09 10:15:09 jenkins-squash scl[11049]: raise JSONDecodeError("Expecting value", s, err.value) from None
      Feb 09 10:15:09 jenkins-squash scl[11049]: json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
      Feb 09 10:15:09 jenkins-squash scl[11049]: 2017-02-09 10:15:09,548 200 GET /metrics/autoload.js?bokeh-autoload-element=c102b89e-b67a-49eb-a38a-d73b3f48e69b (67.212.196.101) 3.65ms
      Feb 09 10:15:09 jenkins-squash oauth2_proxy[10475]: 67.212.196.101 - josh@hoblitt.com [09/Feb/2017:10:15:09 -0800] jhoblitt-larry-bokeh.lsst.codes GET 127.0.0.1:5006 "/metrics/autoload.js?bokeh-autoload-element=c102b89e-b67a-49eb-a38a-d73b3f48e69b" HTTP/1.0 "Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:51.0) Gecko/20100101 Firefox/51.0" 200 3675 0.005
      Feb 09 10:15:10 jenkins-squash oauth2_proxy[10475]: 67.212.196.101 - - [09/Feb/2017:10:14:55 -0800] jhoblitt-larry-bokeh.lsst.codes GET - "/metrics/ws?bokeh-protocol-version=1.0&bokeh-session-id=2poW0vQDLZA6p6C0Q1bsmdwjuGTJ2uRjHpzH28EgsHxa" HTTP/1.0 "Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:51.0) Gecko/20100101 Firefox/51.0" 0 0 14.960
      Feb 09 10:15:10 jenkins-squash scl[11049]: 2017-02-09 10:15:10,663 WebSocket connection opened
      Feb 09 10:15:10 jenkins-squash scl[11049]: 2017-02-09 10:15:10,663 ServerConnection created
      
      

        Attachments

          Issue Links

            Activity

            jhoblitt Joshua Hoblitt created issue -
            jhoblitt Joshua Hoblitt made changes -
            Field Original Value New Value
            Epic Link DM-8478 [ 28110 ]
            jhoblitt Joshua Hoblitt made changes -
            Link This issue blocks DM-8794 [ DM-8794 ]
            afausti Angelo Fausti made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            afausti Angelo Fausti made changes -
            Story Points 1.4
            Hide
            afausti Angelo Fausti added a comment - - edited

            Database migrations were lost during the transition from sqlite to mariadb when the initial migration was recreated by mistake.

            The old migrations were recovered and a new migration 0004_auto_20170214_1411.py was created using the latest dump of the QA production database. They were pushed to the squah production branch:

            https://github.com/lsst-sqre/qa-dashboard/tree/production/squash/dashboard/migrations

            tested both locally and on a deployed environment.

            (venv)[vagrant@jenkins-squash squash]$ python manage.py migrate
            Operations to perform:
              Synchronize unmigrated apps: messages, rest_framework, staticfiles
              Apply all migrations: auth, sessions, dashboard, admin, authtoken, contenttypes
            Synchronizing apps without migrations:
              Creating tables...
                Running deferred SQL...
              Installing custom SQL...
            Running migrations:
              Rendering model states... DONE
              Applying dashboard.0004_auto_20170214_1411... OK
            

            As a result we have a new procedure for validating database migrations as proposed here: https://github.com/lsst-sqre/sandbox-jenkins-demo/pull/68

            Show
            afausti Angelo Fausti added a comment - - edited Database migrations were lost during the transition from sqlite to mariadb when the initial migration was recreated by mistake. The old migrations were recovered and a new migration 0004_auto_20170214_1411.py was created using the latest dump of the QA production database. They were pushed to the squah production branch: https://github.com/lsst-sqre/qa-dashboard/tree/production/squash/dashboard/migrations tested both locally and on a deployed environment. (venv)[vagrant@jenkins-squash squash]$ python manage.py migrate Operations to perform: Synchronize unmigrated apps: messages, rest_framework, staticfiles Apply all migrations: auth, sessions, dashboard, admin, authtoken, contenttypes Synchronizing apps without migrations: Creating tables... Running deferred SQL... Installing custom SQL... Running migrations: Rendering model states... DONE Applying dashboard.0004_auto_20170214_1411... OK As a result we have a new procedure for validating database migrations as proposed here: https://github.com/lsst-sqre/sandbox-jenkins-demo/pull/68
            afausti Angelo Fausti made changes -
            Reviewers Joshua Hoblitt [ jhoblitt ]
            Status In Progress [ 3 ] In Review [ 10004 ]
            Hide
            jhoblitt Joshua Hoblitt added a comment -

            Angelo Fausti Could you open a PR that incorporates your comments on https://github.com/lsst-sqre/sandbox-jenkins-demo/pull/68 ?

            Show
            jhoblitt Joshua Hoblitt added a comment - Angelo Fausti Could you open a PR that incorporates your comments on https://github.com/lsst-sqre/sandbox-jenkins-demo/pull/68 ?
            Hide
            afausti Angelo Fausti added a comment -

            Sure will do that.

            Show
            afausti Angelo Fausti added a comment - Sure will do that.
            Hide
            afausti Angelo Fausti added a comment -
            Show
            afausti Angelo Fausti added a comment - Here is the new PR https://github.com/lsst-sqre/sandbox-jenkins-demo/pull/70
            Hide
            afausti Angelo Fausti added a comment -

            As part of this ticket also added missing migrations back to master, and added instructions to validade migrations locally.

            https://github.com/lsst-sqre/qa-dashboard/blob/master/squash/dashboard/migrations/README.md

            Show
            afausti Angelo Fausti added a comment - As part of this ticket also added missing migrations back to master, and added instructions to validade migrations locally. https://github.com/lsst-sqre/qa-dashboard/blob/master/squash/dashboard/migrations/README.md
            Hide
            jhoblitt Joshua Hoblitt added a comment -

            Tested as working in a sandbox env and successfully deployed to production.

            Show
            jhoblitt Joshua Hoblitt added a comment - Tested as working in a sandbox env and successfully deployed to production.
            jhoblitt Joshua Hoblitt made changes -
            Status In Review [ 10004 ] Reviewed [ 10101 ]
            afausti Angelo Fausti made changes -
            Resolution Done [ 10000 ]
            Status Reviewed [ 10101 ] Done [ 10002 ]

              People

              • Assignee:
                afausti Angelo Fausti
                Reporter:
                jhoblitt Joshua Hoblitt
                Reviewers:
                Joshua Hoblitt
                Watchers:
                Angelo Fausti, Jonathan Sick, Joshua Hoblitt
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel