Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-7005

List the code changes from job to job and link the package names with the corresponding git url

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Motivated by the deviation seen from build 156 to 157 in https://squash.lsst.codes/AM1 (caused by a commit in meas_algorithms package), and other deviations seen later as well, we want to show the list of packages that changed in the current build with respect to the previous build and link to the the git commit URLs to easily identify what caused that deviation.

        Attachments

          Activity

          Hide
          afausti Angelo Fausti added a comment -

          Logged more 6 SP for week Oct 31 - Nov 4

          • Return packages that changed with respect to the previous job in the measurements serializer, this implementation uses python Set to get the package differences
          • Changed get_meas_by_dataset_and_metric() in services.py to get list os package names and corresponding git urls
          • Use bokeh HTMLTemplateFormatter to link package names with the git urls in the metrics app data table (Package column)
          • Added more test data

          Final implementation for this ticket is available here:
          https://angelo-squash-squash.lsst.codes

          Show
          afausti Angelo Fausti added a comment - Logged more 6 SP for week Oct 31 - Nov 4 Return packages that changed with respect to the previous job in the measurements serializer, this implementation uses python Set to get the package differences Changed get_meas_by_dataset_and_metric() in services.py to get list os package names and corresponding git urls Use bokeh HTMLTemplateFormatter to link package names with the git urls in the metrics app data table (Package column) Added more test data Final implementation for this ticket is available here: https://angelo-squash-squash.lsst.codes
          Hide
          afausti Angelo Fausti added a comment -

          The implementation is available for testing at

          https://angelo-squash-squash.lsst.codes

          Look at the measurements API endpoint and notice that it returns just the data consumed by the metrics app then try the interaction between the data points in the plot and in the data table.

          Show
          afausti Angelo Fausti added a comment - The implementation is available for testing at https://angelo-squash-squash.lsst.codes Look at the measurements API endpoint and notice that it returns just the data consumed by the metrics app then try the interaction between the data points in the plot and in the data table.
          Hide
          afausti Angelo Fausti added a comment -

          I think I've found the right solution to connect the measurements (dots in the time series plot) with the data table listing the code changes. The bokeh data table has a nice feature called " scroll_to_selection" so that if you select the data point by any method the corresponding value is highlighted in the table. So if we change the behaviour of the "Tap Select" tool to just select the data points it works nicely. Then instead of open the corresponding Jenkins URL on Tap, the ci_url can be linked from the table by clicking on the corresponding "Job ID" and of course the package name can be linked to the github commit URL

          Show
          afausti Angelo Fausti added a comment - I think I've found the right solution to connect the measurements (dots in the time series plot) with the data table listing the code changes. The bokeh data table has a nice feature called " scroll_to_selection" so that if you select the data point by any method the corresponding value is highlighted in the table. So if we change the behaviour of the "Tap Select" tool to just select the data points it works nicely. Then instead of open the corresponding Jenkins URL on Tap, the ci_url can be linked from the table by clicking on the corresponding "Job ID" and of course the package name can be linked to the github commit URL
          Hide
          afausti Angelo Fausti added a comment -

          Logged more 6 SP for week Oct 17-21

          Show
          afausti Angelo Fausti added a comment - Logged more 6 SP for week Oct 17-21
          Hide
          afausti Angelo Fausti added a comment - - edited
          • finished (need real test though) a method to return the list of packages that changed in a given job with respect to the previous one (check diffs in the git commit sha of each package) the measurements api returns the required info for those packages

          {
              "count": 6,
              "next": null,
              "previous": null,
              "results": [
                  {
                      "value": 4.0,
                      "ci_id": "1",
                      "ci_url": "https://ci.lsst.codes/job/ci_cfht/1/",
                      "date": "2016-09-15T00:00:00.001Z",
                      "changed_packages": {
                          "previous_job": [
                              {
                                  "git_url": "https://github.com/lsst/afw.git",
                                  "name": "afw",
                                  "git_commit": "fc355a99abe3425003b0e5fbe1e13a39644b1e95",
                                  "ci_id": "1",
                                  "build_version": "b2000"
                              }
                          ],
                          "current_job": [
                              {
                                  "git_url": "https://github.com/lsst/afw.git",
                                  "name": "afw",
                                  "git_commit": "fc355a99abe3425003b0e5fbe1e13a39644b1e95",
                                  "ci_id": "1",
                                  "build_version": "b2000"
                              }
                          ]
                      }
                  },
                  {
                      "value": 6.0,
                      "ci_id": "2",
                      "ci_url": "https://ci.lsst.codes/job/ci_cfht/2/",
                      "date": "2016-09-15T08:00:00.001Z",
                      "changed_packages": {
                          "previous_job": [
                              {
                                  "git_url": "https://github.com/lsst/afw.git",
                                  "name": "afw",
                                  "git_commit": "fc355a99abe3425003b0e5fbe1e13a39644b1e95",
                                  "ci_id": "1",
                                  "build_version": "b2000"
                              }
                          ],
                          "current_job": [
                              {
                                  "git_url": "https://github.com/lsst/afw.git",
                                  "name": "afw",
                                  "git_commit": "fc355a99abe3425003b0e5fbe1e13a39644b1e95",
                                  "ci_id": "2",
                                  "build_version": "b2000"
                              }
                          ]
                      }
                  },
          
          

          Show
          afausti Angelo Fausti added a comment - - edited finished (need real test though) a method to return the list of packages that changed in a given job with respect to the previous one (check diffs in the git commit sha of each package) the measurements api returns the required info for those packages { "count": 6, "next": null, "previous": null, "results": [ { "value": 4.0, "ci_id": "1", "ci_url": "https://ci.lsst.codes/job/ci_cfht/1/", "date": "2016-09-15T00:00:00.001Z", "changed_packages": { "previous_job": [ { "git_url": "https://github.com/lsst/afw.git", "name": "afw", "git_commit": "fc355a99abe3425003b0e5fbe1e13a39644b1e95", "ci_id": "1", "build_version": "b2000" } ], "current_job": [ { "git_url": "https://github.com/lsst/afw.git", "name": "afw", "git_commit": "fc355a99abe3425003b0e5fbe1e13a39644b1e95", "ci_id": "1", "build_version": "b2000" } ] } }, { "value": 6.0, "ci_id": "2", "ci_url": "https://ci.lsst.codes/job/ci_cfht/2/", "date": "2016-09-15T08:00:00.001Z", "changed_packages": { "previous_job": [ { "git_url": "https://github.com/lsst/afw.git", "name": "afw", "git_commit": "fc355a99abe3425003b0e5fbe1e13a39644b1e95", "ci_id": "1", "build_version": "b2000" } ], "current_job": [ { "git_url": "https://github.com/lsst/afw.git", "name": "afw", "git_commit": "fc355a99abe3425003b0e5fbe1e13a39644b1e95", "ci_id": "2", "build_version": "b2000" } ] } }, implemented bokeh data table to present the results (any suggestion here?) still learning how to update the bokeh data table connected with the hover tool, that is possible using a custom JS callback as explained here http://bokeh.pydata.org/en/latest/docs/user_guide/interaction/callbacks.html#customjs-for-hover updated bokeh to the newest version 0.12.3 there is a css issue with bootstrap and bokeh data tables (pending)

            People

            • Assignee:
              afausti Angelo Fausti
              Reporter:
              afausti Angelo Fausti
              Watchers:
              Angelo Fausti
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Summary Panel