Joshua Hoblitt added a comment - 07/Nov/16 3:17 PM I'm seeing a new error from the bokeh daemon when bringing up a clean test env, this is after running python manage.py loaddata initial_data Nov 07 14 : 10 : 13 jenkins-squash bokeh[ 15063 ]: 2016 - 11 - 07 14 : 10 : 13 , 140 Error running application handler <bokeh.application.handlers.script.ScriptHandler object at 0x3c7fd10 >: 'dates' Nov 07 14 : 10 : 13 jenkins-squash bokeh[ 15063 ]: File "metrics.py" , line 150 , in update_data_source: Nov 07 14 : 10 : 13 jenkins-squash bokeh[ 15063 ]: size = len(self.data[ 'dates' ]) Traceback (most recent call last): Nov 07 14 : 10 : 13 jenkins-squash bokeh[ 15063 ]: File "/opt/apps/qa-dashboard/venv/lib/python2.7/site-packages/bokeh/application/handlers/code_runner.py" , line 83 , in run Nov 07 14 : 10 : 13 jenkins-squash bokeh[ 15063 ]: exec(self._code, module.__dict__) Nov 07 14 : 10 : 13 jenkins-squash bokeh[ 15063 ]: File "/opt/apps/qa-dashboard/squash/dashboard/viz/metrics.py" , line 319 , in <module> Nov 07 14 : 10 : 13 jenkins-squash bokeh[ 15063 ]: curdoc().add_root(Metrics().layout) Nov 07 14 : 10 : 13 jenkins-squash bokeh[ 15063 ]: File "/opt/apps/qa-dashboard/squash/dashboard/viz/metrics.py" , line 29 , in __init__ Nov 07 14 : 10 : 13 jenkins-squash bokeh[ 15063 ]: self.compose_layout() Nov 07 14 : 10 : 13 jenkins-squash bokeh[ 15063 ]: File "/opt/apps/qa-dashboard/squash/dashboard/viz/metrics.py" , line 62 , in compose_layout Nov 07 14 : 10 : 13 jenkins-squash bokeh[ 15063 ]: self.update_data_source() Nov 07 14 : 10 : 13 jenkins-squash bokeh[ 15063 ]: File "/opt/apps/qa-dashboard/squash/dashboard/viz/metrics.py" , line 150 , in update_data_source Nov 07 14 : 10 : 13 jenkins-squash bokeh[ 15063 ]: size = len(self.data[ 'dates' ]) Nov 07 14 : 10 : 13 jenkins-squash bokeh[ 15063 ]: KeyError: 'dates' The KPMs page is also blank.

Angelo Fausti added a comment - 07/Nov/16 3:46 PM - edited Just fixed to make sure the metrics app does not break when there is no data in the db. Note that in the README instructions there is a new command to load test data for development and testing environments. That's why I didn't catch this bug earlier, I followed my own instructions and loaded the test data before starting the bokeh server.

Joshua Hoblitt added a comment - 16/Nov/16 10:42 AM I have merged my branch that adds back MySQL-python and retested from master. The error message from before is gone, however, validate_drp was failing due to the API rejecting the call. It appears that a new version of post-qa ( 1.2.0 was tagged/released. I have tried testing with that. It appears to be missing a dep on functools32 . After installing that package manually, post-qa is failing with a json validation error. { 'regex' : (<function is_regex at 0x22e4b90 >, < class 'sre_constants.error' >), 'hostname' : (<function is_host_name at 0x22c45f0 >, ()), 'uri' : (<function is_uri at 0x22c4668 >, <type 'exceptions.ValueError' >), 'ipv6' : (<function is_ipv6 at 0x22c4578 >, < class 'socket.error' >), 'host-name' : (<function is_host_name at 0x22c45f0 >, ()), 'ipv4' : (<function is_ipv4 at 0x22c4500 >, ()), 'ip-address' : (<function is_ipv4 at 0x22c4500 >, ()), 'time' : (<function is_time at 0x22e4aa0 >, <type 'exceptions.ValueError' >), 'date' : (<function is_date at 0x22e4b18 >, <type 'exceptions.ValueError' >), 'email' : (<function is_email at 0x22c4488 >, ()), 'date-time' : (<function is_date at 0x22e4c08 >, ())} Traceback (most recent call last): File "/home/jenkins-slave/workspace/validate_drp/dataset/cfht/label/centos-7/python/py2/post-qa/venv/bin/post-qa" , line 9 , in <module> load_entry_point( 'post-qa==1.2.0' , 'console_scripts' , 'post-qa' )() File "/home/jenkins-slave/workspace/validate_drp/dataset/cfht/label/centos-7/python/py2/post-qa/venv/lib/python2.7/site-packages/postqa/cli.py" , line 25 , in run_post_qa job_json = build_job_json(args.qa_json_path, args.lsstsw_dirname) File "/home/jenkins-slave/workspace/validate_drp/dataset/cfht/label/centos-7/python/py2/post-qa/venv/lib/python2.7/site-packages/postqa/cli.py" , line 92 , in build_job_json validate(job_json) File "/home/jenkins-slave/workspace/validate_drp/dataset/cfht/label/centos-7/python/py2/post-qa/venv/lib/python2.7/site-packages/postqa/schemas.py" , line 23 , in validate jsonschema.validate(doc, schema, format_checker=format_checker) File "/home/jenkins-slave/workspace/validate_drp/dataset/cfht/label/centos-7/python/py2/post-qa/venv/lib/python2.7/site-packages/jsonschema/validators.py" , line 478 , in validate cls(schema, *args, **kwargs).validate(instance) File "/home/jenkins-slave/workspace/validate_drp/dataset/cfht/label/centos-7/python/py2/post-qa/venv/lib/python2.7/site-packages/jsonschema/validators.py" , line 123 , in validate raise error jsonschema.exceptions.ValidationError: {u 'units' : u 'milliarcsecond' , u 'description' : u 'Median RMS of the astrometric distance distribution for stellar pairs with separation of D arcmin (repeatability)' , u 'value' : 6.712600663440999 , u 'label' : u 'AM1' } is not valid under any of the given schemas   Failed validating u 'oneOf' in schema[u 'properties' ][u 'measurements' ][u 'items' ][u 'properties' ][u 'value' ]: {u 'oneOf' : [{u 'type' : u 'number' }, {u 'type' : u 'null' }]}   On instance[u 'measurements' ][ 0 ][u 'value' ]: {u 'description' : u 'Median RMS of the astrometric distance distribution for stellar pairs with separation of D arcmin (repeatability)' , u 'label' : u 'AM1' , u 'units' : u 'milliarcsecond' , u 'value' : 6.712600663440999 } Jonathan Sick are there unmerged changes to validate_drp needed for the new version of post-qa ?

Jonathan Sick added a comment - 16/Nov/16 12:03 PM I'm looking at the validation and dependency issues in DM-8285 ; meanwhile is should be entirely possible to continue to use post-qa v1.1.0 or v1.1.1 assuming that there are no changes to the SQUASH REST API.

Angelo Fausti added a comment - 16/Nov/16 12:25 PM Jonathan Sick , Joshua Hoblitt Indeed there was a change in the measurements serializer that propagated to the job serializer which is nested. It is clear that we need more tests. post-qa has a schema validation for the jobs api endpoint and that would have spot this change. I have applied the fix to this ticket branch and merged to master.

Joshua Hoblitt added a comment - 18/Nov/16 7:54 AM The new master branch of qa-dashboard was able to accept input from post-qa 1.1.0 . I updated the production instance last night. Per discussion on slack with Angelo Fausti , the deployment appears successful.