Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-8195

deploy squash code changes feature

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      DM-7005 implements a data table connected with the metrics plot that list the jobs, measurement values and packages that changed with respect to the previous job linked to the corresponding git urls.

      The implementation can be tested here:
      https://angelo-squash-squash.lsst.codes/

      it was merged to master and can be shipped to production.

        Attachments

          Issue Links

            Activity

            Hide
            jhoblitt Joshua Hoblitt added a comment -

            I'm seeing a new error from the bokeh daemon when bringing up a clean test env, this is after running python manage.py loaddata initial_data

            Nov 07 14:10:13 jenkins-squash bokeh[15063]: 2016-11-07 14:10:13,140 Error running application handler <bokeh.application.handlers.script.ScriptHandler object at 0x3c7fd10>: 'dates'
            Nov 07 14:10:13 jenkins-squash bokeh[15063]: File "metrics.py", line 150, in update_data_source:
            Nov 07 14:10:13 jenkins-squash bokeh[15063]: size = len(self.data['dates']) Traceback (most recent call last):
            Nov 07 14:10:13 jenkins-squash bokeh[15063]: File "/opt/apps/qa-dashboard/venv/lib/python2.7/site-packages/bokeh/application/handlers/code_runner.py", line 83, in run
            Nov 07 14:10:13 jenkins-squash bokeh[15063]: exec(self._code, module.__dict__)
            Nov 07 14:10:13 jenkins-squash bokeh[15063]: File "/opt/apps/qa-dashboard/squash/dashboard/viz/metrics.py", line 319, in <module>
            Nov 07 14:10:13 jenkins-squash bokeh[15063]: curdoc().add_root(Metrics().layout)
            Nov 07 14:10:13 jenkins-squash bokeh[15063]: File "/opt/apps/qa-dashboard/squash/dashboard/viz/metrics.py", line 29, in __init__
            Nov 07 14:10:13 jenkins-squash bokeh[15063]: self.compose_layout()
            Nov 07 14:10:13 jenkins-squash bokeh[15063]: File "/opt/apps/qa-dashboard/squash/dashboard/viz/metrics.py", line 62, in compose_layout
            Nov 07 14:10:13 jenkins-squash bokeh[15063]: self.update_data_source()
            Nov 07 14:10:13 jenkins-squash bokeh[15063]: File "/opt/apps/qa-dashboard/squash/dashboard/viz/metrics.py", line 150, in update_data_source
            Nov 07 14:10:13 jenkins-squash bokeh[15063]: size = len(self.data['dates'])
            Nov 07 14:10:13 jenkins-squash bokeh[15063]: KeyError: 'dates'
            

            The KPMs page is also blank.

            Show
            jhoblitt Joshua Hoblitt added a comment - I'm seeing a new error from the bokeh daemon when bringing up a clean test env, this is after running python manage.py loaddata initial_data Nov 07 14 : 10 : 13 jenkins-squash bokeh[ 15063 ]: 2016 - 11 - 07 14 : 10 : 13 , 140 Error running application handler <bokeh.application.handlers.script.ScriptHandler object at 0x3c7fd10 >: 'dates' Nov 07 14 : 10 : 13 jenkins-squash bokeh[ 15063 ]: File "metrics.py" , line 150 , in update_data_source: Nov 07 14 : 10 : 13 jenkins-squash bokeh[ 15063 ]: size = len(self.data[ 'dates' ]) Traceback (most recent call last): Nov 07 14 : 10 : 13 jenkins-squash bokeh[ 15063 ]: File "/opt/apps/qa-dashboard/venv/lib/python2.7/site-packages/bokeh/application/handlers/code_runner.py" , line 83 , in run Nov 07 14 : 10 : 13 jenkins-squash bokeh[ 15063 ]: exec(self._code, module.__dict__) Nov 07 14 : 10 : 13 jenkins-squash bokeh[ 15063 ]: File "/opt/apps/qa-dashboard/squash/dashboard/viz/metrics.py" , line 319 , in <module> Nov 07 14 : 10 : 13 jenkins-squash bokeh[ 15063 ]: curdoc().add_root(Metrics().layout) Nov 07 14 : 10 : 13 jenkins-squash bokeh[ 15063 ]: File "/opt/apps/qa-dashboard/squash/dashboard/viz/metrics.py" , line 29 , in __init__ Nov 07 14 : 10 : 13 jenkins-squash bokeh[ 15063 ]: self.compose_layout() Nov 07 14 : 10 : 13 jenkins-squash bokeh[ 15063 ]: File "/opt/apps/qa-dashboard/squash/dashboard/viz/metrics.py" , line 62 , in compose_layout Nov 07 14 : 10 : 13 jenkins-squash bokeh[ 15063 ]: self.update_data_source() Nov 07 14 : 10 : 13 jenkins-squash bokeh[ 15063 ]: File "/opt/apps/qa-dashboard/squash/dashboard/viz/metrics.py" , line 150 , in update_data_source Nov 07 14 : 10 : 13 jenkins-squash bokeh[ 15063 ]: size = len(self.data[ 'dates' ]) Nov 07 14 : 10 : 13 jenkins-squash bokeh[ 15063 ]: KeyError: 'dates' The KPMs page is also blank.
            Hide
            afausti Angelo Fausti added a comment - - edited

            Just fixed to make sure the metrics app does not break when there is no data in the db.

            Note that in the README instructions there is a new command to load test data for development and testing environments. That's why I didn't catch this bug earlier, I followed my own instructions and loaded the test data before starting the bokeh server.

            Show
            afausti Angelo Fausti added a comment - - edited Just fixed to make sure the metrics app does not break when there is no data in the db. Note that in the README instructions there is a new command to load test data for development and testing environments. That's why I didn't catch this bug earlier, I followed my own instructions and loaded the test data before starting the bokeh server.
            Hide
            jhoblitt Joshua Hoblitt added a comment -

            I have merged my branch that adds back MySQL-python and retested from master. The error message from before is gone, however, validate_drp was failing due to the API rejecting the call. It appears that a new version of post-qa (1.2.0 was tagged/released. I have tried testing with that. It appears to be missing a dep on functools32. After installing that package manually, post-qa is failing with a json validation error.

            {'regex': (<function is_regex at 0x22e4b90>, <class 'sre_constants.error'>), 'hostname': (<function is_host_name at 0x22c45f0>, ()), 'uri': (<function is_uri at 0x22c4668>, <type 'exceptions.ValueError'>), 'ipv6': (<function is_ipv6 at 0x22c4578>, <class 'socket.error'>), 'host-name': (<function is_host_name at 0x22c45f0>, ()), 'ipv4': (<function is_ipv4 at 0x22c4500>, ()), 'ip-address': (<function is_ipv4 at 0x22c4500>, ()), 'time': (<function is_time at 0x22e4aa0>, <type 'exceptions.ValueError'>), 'date': (<function is_date at 0x22e4b18>, <type 'exceptions.ValueError'>), 'email': (<function is_email at 0x22c4488>, ()), 'date-time': (<function is_date at 0x22e4c08>, ())}
            Traceback (most recent call last):
              File "/home/jenkins-slave/workspace/validate_drp/dataset/cfht/label/centos-7/python/py2/post-qa/venv/bin/post-qa", line 9, in <module>
                load_entry_point('post-qa==1.2.0', 'console_scripts', 'post-qa')()
              File "/home/jenkins-slave/workspace/validate_drp/dataset/cfht/label/centos-7/python/py2/post-qa/venv/lib/python2.7/site-packages/postqa/cli.py", line 25, in run_post_qa
                job_json = build_job_json(args.qa_json_path, args.lsstsw_dirname)
              File "/home/jenkins-slave/workspace/validate_drp/dataset/cfht/label/centos-7/python/py2/post-qa/venv/lib/python2.7/site-packages/postqa/cli.py", line 92, in build_job_json
                validate(job_json)
              File "/home/jenkins-slave/workspace/validate_drp/dataset/cfht/label/centos-7/python/py2/post-qa/venv/lib/python2.7/site-packages/postqa/schemas.py", line 23, in validate
                jsonschema.validate(doc, schema, format_checker=format_checker)
              File "/home/jenkins-slave/workspace/validate_drp/dataset/cfht/label/centos-7/python/py2/post-qa/venv/lib/python2.7/site-packages/jsonschema/validators.py", line 478, in validate
                cls(schema, *args, **kwargs).validate(instance)
              File "/home/jenkins-slave/workspace/validate_drp/dataset/cfht/label/centos-7/python/py2/post-qa/venv/lib/python2.7/site-packages/jsonschema/validators.py", line 123, in validate
                raise error
            jsonschema.exceptions.ValidationError: {u'units': u'milliarcsecond', u'description': u'Median RMS of the astrometric distance distribution for stellar pairs with separation of D arcmin (repeatability)', u'value': 6.712600663440999, u'label': u'AM1'} is not valid under any of the given schemas
             
            Failed validating u'oneOf' in schema[u'properties'][u'measurements'][u'items'][u'properties'][u'value']:
                {u'oneOf': [{u'type': u'number'}, {u'type': u'null'}]}
             
            On instance[u'measurements'][0][u'value']:
                {u'description': u'Median RMS of the astrometric distance distribution for stellar pairs with separation of D arcmin (repeatability)',
                 u'label': u'AM1',
                 u'units': u'milliarcsecond',
                 u'value': 6.712600663440999}
            

            Jonathan Sick are there unmerged changes to validate_drp needed for the new version of post-qa?

            Show
            jhoblitt Joshua Hoblitt added a comment - I have merged my branch that adds back MySQL-python and retested from master. The error message from before is gone, however, validate_drp was failing due to the API rejecting the call. It appears that a new version of post-qa ( 1.2.0 was tagged/released. I have tried testing with that. It appears to be missing a dep on functools32 . After installing that package manually, post-qa is failing with a json validation error. { 'regex' : (<function is_regex at 0x22e4b90 >, < class 'sre_constants.error' >), 'hostname' : (<function is_host_name at 0x22c45f0 >, ()), 'uri' : (<function is_uri at 0x22c4668 >, <type 'exceptions.ValueError' >), 'ipv6' : (<function is_ipv6 at 0x22c4578 >, < class 'socket.error' >), 'host-name' : (<function is_host_name at 0x22c45f0 >, ()), 'ipv4' : (<function is_ipv4 at 0x22c4500 >, ()), 'ip-address' : (<function is_ipv4 at 0x22c4500 >, ()), 'time' : (<function is_time at 0x22e4aa0 >, <type 'exceptions.ValueError' >), 'date' : (<function is_date at 0x22e4b18 >, <type 'exceptions.ValueError' >), 'email' : (<function is_email at 0x22c4488 >, ()), 'date-time' : (<function is_date at 0x22e4c08 >, ())} Traceback (most recent call last): File "/home/jenkins-slave/workspace/validate_drp/dataset/cfht/label/centos-7/python/py2/post-qa/venv/bin/post-qa" , line 9 , in <module> load_entry_point( 'post-qa==1.2.0' , 'console_scripts' , 'post-qa' )() File "/home/jenkins-slave/workspace/validate_drp/dataset/cfht/label/centos-7/python/py2/post-qa/venv/lib/python2.7/site-packages/postqa/cli.py" , line 25 , in run_post_qa job_json = build_job_json(args.qa_json_path, args.lsstsw_dirname) File "/home/jenkins-slave/workspace/validate_drp/dataset/cfht/label/centos-7/python/py2/post-qa/venv/lib/python2.7/site-packages/postqa/cli.py" , line 92 , in build_job_json validate(job_json) File "/home/jenkins-slave/workspace/validate_drp/dataset/cfht/label/centos-7/python/py2/post-qa/venv/lib/python2.7/site-packages/postqa/schemas.py" , line 23 , in validate jsonschema.validate(doc, schema, format_checker=format_checker) File "/home/jenkins-slave/workspace/validate_drp/dataset/cfht/label/centos-7/python/py2/post-qa/venv/lib/python2.7/site-packages/jsonschema/validators.py" , line 478 , in validate cls(schema, *args, **kwargs).validate(instance) File "/home/jenkins-slave/workspace/validate_drp/dataset/cfht/label/centos-7/python/py2/post-qa/venv/lib/python2.7/site-packages/jsonschema/validators.py" , line 123 , in validate raise error jsonschema.exceptions.ValidationError: {u 'units' : u 'milliarcsecond' , u 'description' : u 'Median RMS of the astrometric distance distribution for stellar pairs with separation of D arcmin (repeatability)' , u 'value' : 6.712600663440999 , u 'label' : u 'AM1' } is not valid under any of the given schemas   Failed validating u 'oneOf' in schema[u 'properties' ][u 'measurements' ][u 'items' ][u 'properties' ][u 'value' ]: {u 'oneOf' : [{u 'type' : u 'number' }, {u 'type' : u 'null' }]}   On instance[u 'measurements' ][ 0 ][u 'value' ]: {u 'description' : u 'Median RMS of the astrometric distance distribution for stellar pairs with separation of D arcmin (repeatability)' , u 'label' : u 'AM1' , u 'units' : u 'milliarcsecond' , u 'value' : 6.712600663440999 } Jonathan Sick are there unmerged changes to validate_drp needed for the new version of post-qa ?
            Hide
            afausti Angelo Fausti added a comment -

            Joshua Hoblitt for this version of squash I think we need post-qa v1.1.0 from August, but Jonathan Sick can confirm that.

            Show
            afausti Angelo Fausti added a comment - Joshua Hoblitt for this version of squash I think we need post-qa v1.1.0 from August, but Jonathan Sick can confirm that.
            Hide
            jsick Jonathan Sick added a comment -

            I'm looking at the validation and dependency issues in DM-8285; meanwhile is should be entirely possible to continue to use post-qa v1.1.0 or v1.1.1 assuming that there are no changes to the SQUASH REST API.

            Show
            jsick Jonathan Sick added a comment - I'm looking at the validation and dependency issues in DM-8285 ; meanwhile is should be entirely possible to continue to use post-qa v1.1.0 or v1.1.1 assuming that there are no changes to the SQUASH REST API.
            Hide
            afausti Angelo Fausti added a comment -

            Jonathan Sick, Joshua Hoblitt Indeed there was a change in the measurements serializer that propagated to the job serializer which is nested. It is clear that we need more tests. post-qa has a schema validation for the jobs api endpoint and that would have spot this change. I have applied the fix to this ticket branch and merged to master.

            Show
            afausti Angelo Fausti added a comment - Jonathan Sick , Joshua Hoblitt Indeed there was a change in the measurements serializer that propagated to the job serializer which is nested. It is clear that we need more tests. post-qa has a schema validation for the jobs api endpoint and that would have spot this change. I have applied the fix to this ticket branch and merged to master.
            Hide
            jhoblitt Joshua Hoblitt added a comment -

            The new master branch of qa-dashboard was able to accept input from post-qa 1.1.0. I updated the production instance last night. Per discussion on slack with Angelo Fausti, the deployment appears successful.

            Show
            jhoblitt Joshua Hoblitt added a comment - The new master branch of qa-dashboard was able to accept input from post-qa 1.1.0 . I updated the production instance last night. Per discussion on slack with Angelo Fausti , the deployment appears successful.

              People

              Assignee:
              jhoblitt Joshua Hoblitt
              Reporter:
              afausti Angelo Fausti
              Watchers:
              Angelo Fausti, Jonathan Sick, Joshua Hoblitt
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: