Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-7749

Update squash production environment

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      After some improvements in the squash metrics plot it is ready to be deployed to production, this ticket captures that.

      The current master is available in my test environment:

      https://angelo-squash-squash.lsst.codes/

        Attachments

          Activity

          Hide
          jhoblitt Joshua Hoblitt added a comment -

          543ff756f42c9e823eb90d033f4edc8b749f44fb was pushed to prod at ~1500 yesterday and appears to be stable.

          Show
          jhoblitt Joshua Hoblitt added a comment - 543ff756f42c9e823eb90d033f4edc8b749f44fb was pushed to prod at ~1500 yesterday and appears to be stable.
          Hide
          afausti Angelo Fausti added a comment - - edited

          There was a typo data[i+i] it should be data[i+1]!!

          So, the following code works if N<=5 which was the length of my test data set.

           
          last  = len(data) - 2
          for i,value enumerate(data):
             if i < last:     # true if i=1, i=2
                   next = data[i+i]   # 2 + 2 < 5
          
          

          Conclusion, the test dat set was not big enough, adding one more element to the test data set (N=6) would be enough to caught the bug.

          Bug fixed and merged, master should be good to go to production.

          Show
          afausti Angelo Fausti added a comment - - edited There was a typo data [i+i] it should be data [i+1] !! So, the following code works if N<=5 which was the length of my test data set.   last = len (data) - 2 for i,value enumerate (data): if i < last: # true if i=1, i=2 next = data[i + i] # 2 + 2 < 5 Conclusion, the test dat set was not big enough, adding one more element to the test data set (N=6) would be enough to caught the bug. Bug fixed and merged, master should be good to go to production.
          Hide
          afausti Angelo Fausti added a comment -

          @josh seems like a bug in the code that gets the coordinates to draw the box annotations. I could not test it properly with data from production because there was a change in the API https://jira.lsstcorp.org/browse/DM-7517 and I need that to draw correctly the box annotations.

          I will try to reproduce the problem it in my dev environment and fix it.

          Show
          afausti Angelo Fausti added a comment - @josh seems like a bug in the code that gets the coordinates to draw the box annotations. I could not test it properly with data from production because there was a change in the API https://jira.lsstcorp.org/browse/DM-7517 and I need that to draw correctly the box annotations. I will try to reproduce the problem it in my dev environment and fix it.
          Hide
          jhoblitt Joshua Hoblitt added a comment -

          Screenshot of rendering issue.

          Show
          jhoblitt Joshua Hoblitt added a comment - Screenshot of rendering issue.
          Hide
          jhoblitt Joshua Hoblitt added a comment -

          I've rolled back prod to 153dafc2b9bbf3fe7a9403f52b6582708c84d4af and restarted the bokeh and uwsg services in my sandbox. This did not fix the problem in prod but it recovered rendering in the sandbox. It also appears that the first validate_drp run was lost. This is very odd as the deployment did restart these services – perhaps there is some sort of ordering issue? Or a missing migration?.

          In my sandbox, there is a rendering problem with the stretch/etc. lines.

          Show
          jhoblitt Joshua Hoblitt added a comment - I've rolled back prod to 153dafc2b9bbf3fe7a9403f52b6582708c84d4af and restarted the bokeh and uwsg services in my sandbox. This did not fix the problem in prod but it recovered rendering in the sandbox. It also appears that the first validate_drp run was lost. This is very odd as the deployment did restart these services – perhaps there is some sort of ordering issue? Or a missing migration?. In my sandbox, there is a rendering problem with the stretch/etc. lines.
          Hide
          jhoblitt Joshua Hoblitt added a comment -

          Major testing failure. I didn't verify that the data was actually rendered on the KPM page. I'm going to roll prod.

          Sep 27 12:48:50 jenkins-squash oauth2_proxy[20259]: 140.252.90.73 - josh@hoblitt.com [27/Sep/2016:12:48:50 -0700] squash.lsst.codes GET 127.0.0.1:9090 "/dashboard" HTTP/1.0 "Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:48.0) Gecko/20100101 Firefox/48.0" 200 2485 0.008
          Sep 27 12:48:50 jenkins-squash bokeh[22783]: 2016-09-27 12:48:50,400 Error running application handler <bokeh.application.handlers.script.ScriptHandler object at 0x35fbf50>: list index out of range
          Sep 27 12:48:50 jenkins-squash bokeh[22783]: File "metrics.py", line 213, in get_box_coords:
          Sep 27 12:48:50 jenkins-squash bokeh[22783]: next = self.data['ci_id'][i+i] Traceback (most recent call last):
          Sep 27 12:48:50 jenkins-squash bokeh[22783]: File "/opt/apps/qa-dashboard/venv/lib/python2.7/site-packages/bokeh/application/handlers/code_runner.py", line 83, in run
          Sep 27 12:48:50 jenkins-squash bokeh[22783]: exec(self._code, module.__dict__)
          Sep 27 12:48:50 jenkins-squash bokeh[22783]: File "/opt/apps/qa-dashboard/squash/dashboard/viz/metrics.py", line 258, in <module>
          Sep 27 12:48:50 jenkins-squash bokeh[22783]: curdoc().add_root(Metrics().layout)
          Sep 27 12:48:50 jenkins-squash bokeh[22783]: File "/opt/apps/qa-dashboard/squash/dashboard/viz/metrics.py", line 20, in __init__
          Sep 27 12:48:50 jenkins-squash bokeh[22783]: self.compose_layout()
          Sep 27 12:48:50 jenkins-squash bokeh[22783]: File "/opt/apps/qa-dashboard/squash/dashboard/viz/metrics.py", line 58, in compose_layout
          Sep 27 12:48:50 jenkins-squash bokeh[22783]: self.make_plot()
          Sep 27 12:48:50 jenkins-squash bokeh[22783]: File "/opt/apps/qa-dashboard/squash/dashboard/viz/metrics.py", line 179, in make_plot
          Sep 27 12:48:50 jenkins-squash bokeh[22783]: self.box_annotations = self.make_box_annotations()
          Sep 27 12:48:50 jenkins-squash bokeh[22783]: File "/opt/apps/qa-dashboard/squash/dashboard/viz/metrics.py", line 231, in make_box_annotations
          Sep 27 12:48:50 jenkins-squash bokeh[22783]: coords = self.get_box_coords()
          Sep 27 12:48:50 jenkins-squash bokeh[22783]: File "/opt/apps/qa-dashboard/squash/dashboard/viz/metrics.py", line 213, in get_box_coords
          Sep 27 12:48:50 jenkins-squash bokeh[22783]: next = self.data['ci_id'][i+i]
          Sep 27 12:48:50 jenkins-squash bokeh[22783]: IndexError: list index out of range
          

          Show
          jhoblitt Joshua Hoblitt added a comment - Major testing failure. I didn't verify that the data was actually rendered on the KPM page. I'm going to roll prod. Sep 27 12 : 48 : 50 jenkins-squash oauth2_proxy[ 20259 ]: 140.252 . 90.73 - josh @hoblitt .com [ 27 /Sep/ 2016 : 12 : 48 : 50 - 0700 ] squash.lsst.codes GET 127.0 . 0.1 : 9090 "/dashboard" HTTP/ 1.0 "Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:48.0) Gecko/20100101 Firefox/48.0" 200 2485 0.008 Sep 27 12 : 48 : 50 jenkins-squash bokeh[ 22783 ]: 2016 - 09 - 27 12 : 48 : 50 , 400 Error running application handler <bokeh.application.handlers.script.ScriptHandler object at 0x35fbf50 >: list index out of range Sep 27 12 : 48 : 50 jenkins-squash bokeh[ 22783 ]: File "metrics.py" , line 213 , in get_box_coords: Sep 27 12 : 48 : 50 jenkins-squash bokeh[ 22783 ]: next = self.data[ 'ci_id' ][i+i] Traceback (most recent call last): Sep 27 12 : 48 : 50 jenkins-squash bokeh[ 22783 ]: File "/opt/apps/qa-dashboard/venv/lib/python2.7/site-packages/bokeh/application/handlers/code_runner.py" , line 83 , in run Sep 27 12 : 48 : 50 jenkins-squash bokeh[ 22783 ]: exec(self._code, module.__dict__) Sep 27 12 : 48 : 50 jenkins-squash bokeh[ 22783 ]: File "/opt/apps/qa-dashboard/squash/dashboard/viz/metrics.py" , line 258 , in <module> Sep 27 12 : 48 : 50 jenkins-squash bokeh[ 22783 ]: curdoc().add_root(Metrics().layout) Sep 27 12 : 48 : 50 jenkins-squash bokeh[ 22783 ]: File "/opt/apps/qa-dashboard/squash/dashboard/viz/metrics.py" , line 20 , in __init__ Sep 27 12 : 48 : 50 jenkins-squash bokeh[ 22783 ]: self.compose_layout() Sep 27 12 : 48 : 50 jenkins-squash bokeh[ 22783 ]: File "/opt/apps/qa-dashboard/squash/dashboard/viz/metrics.py" , line 58 , in compose_layout Sep 27 12 : 48 : 50 jenkins-squash bokeh[ 22783 ]: self.make_plot() Sep 27 12 : 48 : 50 jenkins-squash bokeh[ 22783 ]: File "/opt/apps/qa-dashboard/squash/dashboard/viz/metrics.py" , line 179 , in make_plot Sep 27 12 : 48 : 50 jenkins-squash bokeh[ 22783 ]: self.box_annotations = self.make_box_annotations() Sep 27 12 : 48 : 50 jenkins-squash bokeh[ 22783 ]: File "/opt/apps/qa-dashboard/squash/dashboard/viz/metrics.py" , line 231 , in make_box_annotations Sep 27 12 : 48 : 50 jenkins-squash bokeh[ 22783 ]: coords = self.get_box_coords() Sep 27 12 : 48 : 50 jenkins-squash bokeh[ 22783 ]: File "/opt/apps/qa-dashboard/squash/dashboard/viz/metrics.py" , line 213 , in get_box_coords Sep 27 12 : 48 : 50 jenkins-squash bokeh[ 22783 ]: next = self.data[ 'ci_id' ][i+i] Sep 27 12 : 48 : 50 jenkins-squash bokeh[ 22783 ]: IndexError: list index out of range
          Hide
          jhoblitt Joshua Hoblitt added a comment -

          I have manually tested that validate_drp is functioning against 78425a1f44ce2eb96086549f7e1b75ce66fe4ca4 in a sandbox env. Proceeding with a prod deploy.

          Show
          jhoblitt Joshua Hoblitt added a comment - I have manually tested that validate_drp is functioning against 78425a1f44ce2eb96086549f7e1b75ce66fe4ca4 in a sandbox env. Proceeding with a prod deploy.
          Show
          jhoblitt Joshua Hoblitt added a comment - Angelo Fausti I presume that you want to deploy https://github.com/lsst-sqre/qa-dashboard/commit/5a543b2777da785fb9f1b751f4aad9366264812a?

            People

            • Assignee:
              jhoblitt Joshua Hoblitt
              Reporter:
              afausti Angelo Fausti
              Watchers:
              Angelo Fausti, Joshua Hoblitt
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Summary Panel