Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-8195

deploy squash code changes feature

    XMLWordPrintable

Details

    • Story
    • Status: Done
    • Resolution: Done
    • None
    • None
    • None

    Description

      DM-7005 implements a data table connected with the metrics plot that list the jobs, measurement values and packages that changed with respect to the previous job linked to the corresponding git urls.

      The implementation can be tested here:
      https://angelo-squash-squash.lsst.codes/

      it was merged to master and can be shipped to production.

      Attachments

        Issue Links

          Activity

            I'm seeing a new error from the bokeh daemon when bringing up a clean test env, this is after running python manage.py loaddata initial_data

            Nov 07 14:10:13 jenkins-squash bokeh[15063]: 2016-11-07 14:10:13,140 Error running application handler <bokeh.application.handlers.script.ScriptHandler object at 0x3c7fd10>: 'dates'
            Nov 07 14:10:13 jenkins-squash bokeh[15063]: File "metrics.py", line 150, in update_data_source:
            Nov 07 14:10:13 jenkins-squash bokeh[15063]: size = len(self.data['dates']) Traceback (most recent call last):
            Nov 07 14:10:13 jenkins-squash bokeh[15063]: File "/opt/apps/qa-dashboard/venv/lib/python2.7/site-packages/bokeh/application/handlers/code_runner.py", line 83, in run
            Nov 07 14:10:13 jenkins-squash bokeh[15063]: exec(self._code, module.__dict__)
            Nov 07 14:10:13 jenkins-squash bokeh[15063]: File "/opt/apps/qa-dashboard/squash/dashboard/viz/metrics.py", line 319, in <module>
            Nov 07 14:10:13 jenkins-squash bokeh[15063]: curdoc().add_root(Metrics().layout)
            Nov 07 14:10:13 jenkins-squash bokeh[15063]: File "/opt/apps/qa-dashboard/squash/dashboard/viz/metrics.py", line 29, in __init__
            Nov 07 14:10:13 jenkins-squash bokeh[15063]: self.compose_layout()
            Nov 07 14:10:13 jenkins-squash bokeh[15063]: File "/opt/apps/qa-dashboard/squash/dashboard/viz/metrics.py", line 62, in compose_layout
            Nov 07 14:10:13 jenkins-squash bokeh[15063]: self.update_data_source()
            Nov 07 14:10:13 jenkins-squash bokeh[15063]: File "/opt/apps/qa-dashboard/squash/dashboard/viz/metrics.py", line 150, in update_data_source
            Nov 07 14:10:13 jenkins-squash bokeh[15063]: size = len(self.data['dates'])
            Nov 07 14:10:13 jenkins-squash bokeh[15063]: KeyError: 'dates'
            

            The KPMs page is also blank.

            jhoblitt Joshua Hoblitt added a comment - I'm seeing a new error from the bokeh daemon when bringing up a clean test env, this is after running python manage.py loaddata initial_data Nov 07 14 : 10 : 13 jenkins-squash bokeh[ 15063 ]: 2016 - 11 - 07 14 : 10 : 13 , 140 Error running application handler <bokeh.application.handlers.script.ScriptHandler object at 0x3c7fd10 >: 'dates' Nov 07 14 : 10 : 13 jenkins-squash bokeh[ 15063 ]: File "metrics.py" , line 150 , in update_data_source: Nov 07 14 : 10 : 13 jenkins-squash bokeh[ 15063 ]: size = len(self.data[ 'dates' ]) Traceback (most recent call last): Nov 07 14 : 10 : 13 jenkins-squash bokeh[ 15063 ]: File "/opt/apps/qa-dashboard/venv/lib/python2.7/site-packages/bokeh/application/handlers/code_runner.py" , line 83 , in run Nov 07 14 : 10 : 13 jenkins-squash bokeh[ 15063 ]: exec(self._code, module.__dict__) Nov 07 14 : 10 : 13 jenkins-squash bokeh[ 15063 ]: File "/opt/apps/qa-dashboard/squash/dashboard/viz/metrics.py" , line 319 , in <module> Nov 07 14 : 10 : 13 jenkins-squash bokeh[ 15063 ]: curdoc().add_root(Metrics().layout) Nov 07 14 : 10 : 13 jenkins-squash bokeh[ 15063 ]: File "/opt/apps/qa-dashboard/squash/dashboard/viz/metrics.py" , line 29 , in __init__ Nov 07 14 : 10 : 13 jenkins-squash bokeh[ 15063 ]: self.compose_layout() Nov 07 14 : 10 : 13 jenkins-squash bokeh[ 15063 ]: File "/opt/apps/qa-dashboard/squash/dashboard/viz/metrics.py" , line 62 , in compose_layout Nov 07 14 : 10 : 13 jenkins-squash bokeh[ 15063 ]: self.update_data_source() Nov 07 14 : 10 : 13 jenkins-squash bokeh[ 15063 ]: File "/opt/apps/qa-dashboard/squash/dashboard/viz/metrics.py" , line 150 , in update_data_source Nov 07 14 : 10 : 13 jenkins-squash bokeh[ 15063 ]: size = len(self.data[ 'dates' ]) Nov 07 14 : 10 : 13 jenkins-squash bokeh[ 15063 ]: KeyError: 'dates' The KPMs page is also blank.
            afausti Angelo Fausti added a comment - - edited

            Just fixed to make sure the metrics app does not break when there is no data in the db.

            Note that in the README instructions there is a new command to load test data for development and testing environments. That's why I didn't catch this bug earlier, I followed my own instructions and loaded the test data before starting the bokeh server.

            afausti Angelo Fausti added a comment - - edited Just fixed to make sure the metrics app does not break when there is no data in the db. Note that in the README instructions there is a new command to load test data for development and testing environments. That's why I didn't catch this bug earlier, I followed my own instructions and loaded the test data before starting the bokeh server.

            I have merged my branch that adds back MySQL-python and retested from master. The error message from before is gone, however, validate_drp was failing due to the API rejecting the call. It appears that a new version of post-qa (1.2.0 was tagged/released. I have tried testing with that. It appears to be missing a dep on functools32. After installing that package manually, post-qa is failing with a json validation error.

            {'regex': (<function is_regex at 0x22e4b90>, <class 'sre_constants.error'>), 'hostname': (<function is_host_name at 0x22c45f0>, ()), 'uri': (<function is_uri at 0x22c4668>, <type 'exceptions.ValueError'>), 'ipv6': (<function is_ipv6 at 0x22c4578>, <class 'socket.error'>), 'host-name': (<function is_host_name at 0x22c45f0>, ()), 'ipv4': (<function is_ipv4 at 0x22c4500>, ()), 'ip-address': (<function is_ipv4 at 0x22c4500>, ()), 'time': (<function is_time at 0x22e4aa0>, <type 'exceptions.ValueError'>), 'date': (<function is_date at 0x22e4b18>, <type 'exceptions.ValueError'>), 'email': (<function is_email at 0x22c4488>, ()), 'date-time': (<function is_date at 0x22e4c08>, ())}
            Traceback (most recent call last):
              File "/home/jenkins-slave/workspace/validate_drp/dataset/cfht/label/centos-7/python/py2/post-qa/venv/bin/post-qa", line 9, in <module>
                load_entry_point('post-qa==1.2.0', 'console_scripts', 'post-qa')()
              File "/home/jenkins-slave/workspace/validate_drp/dataset/cfht/label/centos-7/python/py2/post-qa/venv/lib/python2.7/site-packages/postqa/cli.py", line 25, in run_post_qa
                job_json = build_job_json(args.qa_json_path, args.lsstsw_dirname)
              File "/home/jenkins-slave/workspace/validate_drp/dataset/cfht/label/centos-7/python/py2/post-qa/venv/lib/python2.7/site-packages/postqa/cli.py", line 92, in build_job_json
                validate(job_json)
              File "/home/jenkins-slave/workspace/validate_drp/dataset/cfht/label/centos-7/python/py2/post-qa/venv/lib/python2.7/site-packages/postqa/schemas.py", line 23, in validate
                jsonschema.validate(doc, schema, format_checker=format_checker)
              File "/home/jenkins-slave/workspace/validate_drp/dataset/cfht/label/centos-7/python/py2/post-qa/venv/lib/python2.7/site-packages/jsonschema/validators.py", line 478, in validate
                cls(schema, *args, **kwargs).validate(instance)
              File "/home/jenkins-slave/workspace/validate_drp/dataset/cfht/label/centos-7/python/py2/post-qa/venv/lib/python2.7/site-packages/jsonschema/validators.py", line 123, in validate
                raise error
            jsonschema.exceptions.ValidationError: {u'units': u'milliarcsecond', u'description': u'Median RMS of the astrometric distance distribution for stellar pairs with separation of D arcmin (repeatability)', u'value': 6.712600663440999, u'label': u'AM1'} is not valid under any of the given schemas
             
            Failed validating u'oneOf' in schema[u'properties'][u'measurements'][u'items'][u'properties'][u'value']:
                {u'oneOf': [{u'type': u'number'}, {u'type': u'null'}]}
             
            On instance[u'measurements'][0][u'value']:
                {u'description': u'Median RMS of the astrometric distance distribution for stellar pairs with separation of D arcmin (repeatability)',
                 u'label': u'AM1',
                 u'units': u'milliarcsecond',
                 u'value': 6.712600663440999}
            

            jsick are there unmerged changes to validate_drp needed for the new version of post-qa?

            jhoblitt Joshua Hoblitt added a comment - I have merged my branch that adds back MySQL-python and retested from master. The error message from before is gone, however, validate_drp was failing due to the API rejecting the call. It appears that a new version of post-qa ( 1.2.0 was tagged/released. I have tried testing with that. It appears to be missing a dep on functools32 . After installing that package manually, post-qa is failing with a json validation error. { 'regex' : (<function is_regex at 0x22e4b90 >, < class 'sre_constants.error' >), 'hostname' : (<function is_host_name at 0x22c45f0 >, ()), 'uri' : (<function is_uri at 0x22c4668 >, <type 'exceptions.ValueError' >), 'ipv6' : (<function is_ipv6 at 0x22c4578 >, < class 'socket.error' >), 'host-name' : (<function is_host_name at 0x22c45f0 >, ()), 'ipv4' : (<function is_ipv4 at 0x22c4500 >, ()), 'ip-address' : (<function is_ipv4 at 0x22c4500 >, ()), 'time' : (<function is_time at 0x22e4aa0 >, <type 'exceptions.ValueError' >), 'date' : (<function is_date at 0x22e4b18 >, <type 'exceptions.ValueError' >), 'email' : (<function is_email at 0x22c4488 >, ()), 'date-time' : (<function is_date at 0x22e4c08 >, ())} Traceback (most recent call last): File "/home/jenkins-slave/workspace/validate_drp/dataset/cfht/label/centos-7/python/py2/post-qa/venv/bin/post-qa" , line 9 , in <module> load_entry_point( 'post-qa==1.2.0' , 'console_scripts' , 'post-qa' )() File "/home/jenkins-slave/workspace/validate_drp/dataset/cfht/label/centos-7/python/py2/post-qa/venv/lib/python2.7/site-packages/postqa/cli.py" , line 25 , in run_post_qa job_json = build_job_json(args.qa_json_path, args.lsstsw_dirname) File "/home/jenkins-slave/workspace/validate_drp/dataset/cfht/label/centos-7/python/py2/post-qa/venv/lib/python2.7/site-packages/postqa/cli.py" , line 92 , in build_job_json validate(job_json) File "/home/jenkins-slave/workspace/validate_drp/dataset/cfht/label/centos-7/python/py2/post-qa/venv/lib/python2.7/site-packages/postqa/schemas.py" , line 23 , in validate jsonschema.validate(doc, schema, format_checker=format_checker) File "/home/jenkins-slave/workspace/validate_drp/dataset/cfht/label/centos-7/python/py2/post-qa/venv/lib/python2.7/site-packages/jsonschema/validators.py" , line 478 , in validate cls(schema, *args, **kwargs).validate(instance) File "/home/jenkins-slave/workspace/validate_drp/dataset/cfht/label/centos-7/python/py2/post-qa/venv/lib/python2.7/site-packages/jsonschema/validators.py" , line 123 , in validate raise error jsonschema.exceptions.ValidationError: {u 'units' : u 'milliarcsecond' , u 'description' : u 'Median RMS of the astrometric distance distribution for stellar pairs with separation of D arcmin (repeatability)' , u 'value' : 6.712600663440999 , u 'label' : u 'AM1' } is not valid under any of the given schemas   Failed validating u 'oneOf' in schema[u 'properties' ][u 'measurements' ][u 'items' ][u 'properties' ][u 'value' ]: {u 'oneOf' : [{u 'type' : u 'number' }, {u 'type' : u 'null' }]}   On instance[u 'measurements' ][ 0 ][u 'value' ]: {u 'description' : u 'Median RMS of the astrometric distance distribution for stellar pairs with separation of D arcmin (repeatability)' , u 'label' : u 'AM1' , u 'units' : u 'milliarcsecond' , u 'value' : 6.712600663440999 } jsick are there unmerged changes to validate_drp needed for the new version of post-qa ?

            jhoblitt for this version of squash I think we need post-qa v1.1.0 from August, but jsick can confirm that.

            afausti Angelo Fausti added a comment - jhoblitt for this version of squash I think we need post-qa v1.1.0 from August, but jsick can confirm that.

            I'm looking at the validation and dependency issues in DM-8285; meanwhile is should be entirely possible to continue to use post-qa v1.1.0 or v1.1.1 assuming that there are no changes to the SQUASH REST API.

            jsick Jonathan Sick added a comment - I'm looking at the validation and dependency issues in DM-8285 ; meanwhile is should be entirely possible to continue to use post-qa v1.1.0 or v1.1.1 assuming that there are no changes to the SQUASH REST API.

            jsick, jhoblitt Indeed there was a change in the measurements serializer that propagated to the job serializer which is nested. It is clear that we need more tests. post-qa has a schema validation for the jobs api endpoint and that would have spot this change. I have applied the fix to this ticket branch and merged to master.

            afausti Angelo Fausti added a comment - jsick , jhoblitt Indeed there was a change in the measurements serializer that propagated to the job serializer which is nested. It is clear that we need more tests. post-qa has a schema validation for the jobs api endpoint and that would have spot this change. I have applied the fix to this ticket branch and merged to master.

            The new master branch of qa-dashboard was able to accept input from post-qa 1.1.0. I updated the production instance last night. Per discussion on slack with afausti, the deployment appears successful.

            jhoblitt Joshua Hoblitt added a comment - The new master branch of qa-dashboard was able to accept input from post-qa 1.1.0 . I updated the production instance last night. Per discussion on slack with afausti , the deployment appears successful.

            People

              jhoblitt Joshua Hoblitt
              afausti Angelo Fausti
              Angelo Fausti, Jonathan Sick, Joshua Hoblitt
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.