# Update squash production environment

XMLWordPrintable

## Details

• Type: Story
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s: None
• Labels:
None
• Story Points:
1
• Team:
SQuaRE

## Description

After some improvements in the squash metrics plot it is ready to be deployed to production, this ticket captures that.

The current master is available in my test environment:

## Activity

Show
Joshua Hoblitt added a comment - Angelo Fausti I presume that you want to deploy https://github.com/lsst-sqre/qa-dashboard/commit/5a543b2777da785fb9f1b751f4aad9366264812a?
Hide
Joshua Hoblitt added a comment -

I have manually tested that validate_drp is functioning against 78425a1f44ce2eb96086549f7e1b75ce66fe4ca4 in a sandbox env. Proceeding with a prod deploy.

Show
Joshua Hoblitt added a comment - I have manually tested that validate_drp is functioning against 78425a1f44ce2eb96086549f7e1b75ce66fe4ca4 in a sandbox env. Proceeding with a prod deploy.
Hide
Joshua Hoblitt added a comment -

Major testing failure. I didn't verify that the data was actually rendered on the KPM page. I'm going to roll prod.

 Sep 27 12:48:50 jenkins-squash oauth2_proxy[20259]: 140.252.90.73 - josh@hoblitt.com [27/Sep/2016:12:48:50 -0700] squash.lsst.codes GET 127.0.0.1:9090 "/dashboard" HTTP/1.0 "Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:48.0) Gecko/20100101 Firefox/48.0" 200 2485 0.008 Sep 27 12:48:50 jenkins-squash bokeh[22783]: 2016-09-27 12:48:50,400 Error running application handler : list index out of range Sep 27 12:48:50 jenkins-squash bokeh[22783]: File "metrics.py", line 213, in get_box_coords: Sep 27 12:48:50 jenkins-squash bokeh[22783]: next = self.data['ci_id'][i+i] Traceback (most recent call last): Sep 27 12:48:50 jenkins-squash bokeh[22783]: File "/opt/apps/qa-dashboard/venv/lib/python2.7/site-packages/bokeh/application/handlers/code_runner.py", line 83, in run Sep 27 12:48:50 jenkins-squash bokeh[22783]: exec(self._code, module.__dict__) Sep 27 12:48:50 jenkins-squash bokeh[22783]: File "/opt/apps/qa-dashboard/squash/dashboard/viz/metrics.py", line 258, in  Sep 27 12:48:50 jenkins-squash bokeh[22783]: curdoc().add_root(Metrics().layout) Sep 27 12:48:50 jenkins-squash bokeh[22783]: File "/opt/apps/qa-dashboard/squash/dashboard/viz/metrics.py", line 20, in __init__ Sep 27 12:48:50 jenkins-squash bokeh[22783]: self.compose_layout() Sep 27 12:48:50 jenkins-squash bokeh[22783]: File "/opt/apps/qa-dashboard/squash/dashboard/viz/metrics.py", line 58, in compose_layout Sep 27 12:48:50 jenkins-squash bokeh[22783]: self.make_plot() Sep 27 12:48:50 jenkins-squash bokeh[22783]: File "/opt/apps/qa-dashboard/squash/dashboard/viz/metrics.py", line 179, in make_plot Sep 27 12:48:50 jenkins-squash bokeh[22783]: self.box_annotations = self.make_box_annotations() Sep 27 12:48:50 jenkins-squash bokeh[22783]: File "/opt/apps/qa-dashboard/squash/dashboard/viz/metrics.py", line 231, in make_box_annotations Sep 27 12:48:50 jenkins-squash bokeh[22783]: coords = self.get_box_coords() Sep 27 12:48:50 jenkins-squash bokeh[22783]: File "/opt/apps/qa-dashboard/squash/dashboard/viz/metrics.py", line 213, in get_box_coords Sep 27 12:48:50 jenkins-squash bokeh[22783]: next = self.data['ci_id'][i+i] Sep 27 12:48:50 jenkins-squash bokeh[22783]: IndexError: list index out of range 

Show
Joshua Hoblitt added a comment - Major testing failure. I didn't verify that the data was actually rendered on the KPM page. I'm going to roll prod. Sep 27 12 : 48 : 50 jenkins-squash oauth2_proxy[ 20259 ]: 140.252 . 90.73 - josh @hoblitt .com [ 27 /Sep/ 2016 : 12 : 48 : 50 - 0700 ] squash.lsst.codes GET 127.0 . 0.1 : 9090 "/dashboard" HTTP/ 1.0 "Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:48.0) Gecko/20100101 Firefox/48.0" 200 2485 0.008 Sep 27 12 : 48 : 50 jenkins-squash bokeh[ 22783 ]: 2016 - 09 - 27 12 : 48 : 50 , 400 Error running application handler <bokeh.application.handlers.script.ScriptHandler object at 0x35fbf50 >: list index out of range Sep 27 12 : 48 : 50 jenkins-squash bokeh[ 22783 ]: File "metrics.py" , line 213 , in get_box_coords: Sep 27 12 : 48 : 50 jenkins-squash bokeh[ 22783 ]: next = self.data[ 'ci_id' ][i+i] Traceback (most recent call last): Sep 27 12 : 48 : 50 jenkins-squash bokeh[ 22783 ]: File "/opt/apps/qa-dashboard/venv/lib/python2.7/site-packages/bokeh/application/handlers/code_runner.py" , line 83 , in run Sep 27 12 : 48 : 50 jenkins-squash bokeh[ 22783 ]: exec(self._code, module.__dict__) Sep 27 12 : 48 : 50 jenkins-squash bokeh[ 22783 ]: File "/opt/apps/qa-dashboard/squash/dashboard/viz/metrics.py" , line 258 , in <module> Sep 27 12 : 48 : 50 jenkins-squash bokeh[ 22783 ]: curdoc().add_root(Metrics().layout) Sep 27 12 : 48 : 50 jenkins-squash bokeh[ 22783 ]: File "/opt/apps/qa-dashboard/squash/dashboard/viz/metrics.py" , line 20 , in __init__ Sep 27 12 : 48 : 50 jenkins-squash bokeh[ 22783 ]: self.compose_layout() Sep 27 12 : 48 : 50 jenkins-squash bokeh[ 22783 ]: File "/opt/apps/qa-dashboard/squash/dashboard/viz/metrics.py" , line 58 , in compose_layout Sep 27 12 : 48 : 50 jenkins-squash bokeh[ 22783 ]: self.make_plot() Sep 27 12 : 48 : 50 jenkins-squash bokeh[ 22783 ]: File "/opt/apps/qa-dashboard/squash/dashboard/viz/metrics.py" , line 179 , in make_plot Sep 27 12 : 48 : 50 jenkins-squash bokeh[ 22783 ]: self.box_annotations = self.make_box_annotations() Sep 27 12 : 48 : 50 jenkins-squash bokeh[ 22783 ]: File "/opt/apps/qa-dashboard/squash/dashboard/viz/metrics.py" , line 231 , in make_box_annotations Sep 27 12 : 48 : 50 jenkins-squash bokeh[ 22783 ]: coords = self.get_box_coords() Sep 27 12 : 48 : 50 jenkins-squash bokeh[ 22783 ]: File "/opt/apps/qa-dashboard/squash/dashboard/viz/metrics.py" , line 213 , in get_box_coords Sep 27 12 : 48 : 50 jenkins-squash bokeh[ 22783 ]: next = self.data[ 'ci_id' ][i+i] Sep 27 12 : 48 : 50 jenkins-squash bokeh[ 22783 ]: IndexError: list index out of range
Hide
Joshua Hoblitt added a comment -

I've rolled back prod to 153dafc2b9bbf3fe7a9403f52b6582708c84d4af and restarted the bokeh and uwsg services in my sandbox. This did not fix the problem in prod but it recovered rendering in the sandbox. It also appears that the first validate_drp run was lost. This is very odd as the deployment did restart these services – perhaps there is some sort of ordering issue? Or a missing migration?.

In my sandbox, there is a rendering problem with the stretch/etc. lines.

Show
Joshua Hoblitt added a comment - I've rolled back prod to 153dafc2b9bbf3fe7a9403f52b6582708c84d4af and restarted the bokeh and uwsg services in my sandbox. This did not fix the problem in prod but it recovered rendering in the sandbox. It also appears that the first validate_drp run was lost. This is very odd as the deployment did restart these services – perhaps there is some sort of ordering issue? Or a missing migration?. In my sandbox, there is a rendering problem with the stretch/etc. lines.
Hide
Joshua Hoblitt added a comment -

Screenshot of rendering issue.

Show
Joshua Hoblitt added a comment - Screenshot of rendering issue.
Hide
Angelo Fausti added a comment -

@josh seems like a bug in the code that gets the coordinates to draw the box annotations. I could not test it properly with data from production because there was a change in the API https://jira.lsstcorp.org/browse/DM-7517 and I need that to draw correctly the box annotations.

I will try to reproduce the problem it in my dev environment and fix it.

Show
Angelo Fausti added a comment - @josh seems like a bug in the code that gets the coordinates to draw the box annotations. I could not test it properly with data from production because there was a change in the API https://jira.lsstcorp.org/browse/DM-7517 and I need that to draw correctly the box annotations. I will try to reproduce the problem it in my dev environment and fix it.
Hide
Angelo Fausti added a comment - - edited

There was a typo data[i+i] it should be data[i+1]!!

So, the following code works if N<=5 which was the length of my test data set.

   last = len(data) - 2 for i,value enumerate(data):  if i < last: # true if i=1, i=2  next = data[i+i] # 2 + 2 < 5  

Conclusion, the test dat set was not big enough, adding one more element to the test data set (N=6) would be enough to caught the bug.

Bug fixed and merged, master should be good to go to production.

Show
Angelo Fausti added a comment - - edited There was a typo data [i+i] it should be data [i+1] !! So, the following code works if N<=5 which was the length of my test data set.   last = len (data) - 2 for i,value enumerate (data): if i < last: # true if i=1, i=2 next = data[i + i] # 2 + 2 < 5 Conclusion, the test dat set was not big enough, adding one more element to the test data set (N=6) would be enough to caught the bug. Bug fixed and merged, master should be good to go to production.
Hide
Joshua Hoblitt added a comment -

543ff756f42c9e823eb90d033f4edc8b749f44fb was pushed to prod at ~1500 yesterday and appears to be stable.

Show
Joshua Hoblitt added a comment - 543ff756f42c9e823eb90d033f4edc8b749f44fb was pushed to prod at ~1500 yesterday and appears to be stable.

## People

• Assignee:
Joshua Hoblitt
Reporter:
Angelo Fausti
Watchers:
Angelo Fausti, Joshua Hoblitt