Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-12787

Science Quality Monitoring and Testing

    XMLWordPrintable

    Details

    • Type: Epic
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Epic Name:
      sqre-s18-squash-1
    • Story Points:
      82.5
    • WBS:
      1.02C.10.02
    • Team:
      SQuaRE
    • Cycle:
      Spring 2018

      Description

      This epic focuses on metric monitoring and visualisation both through the SQuaSH harness and the JupyterLab-based Notebook Environment of the LSST Science Platform.

      This epic includes a migration to the Flask API to eliminate DJANGO from our toolchain and make SQuaSH technically consistent with other SQuaRE services

      A major deliverable for S18 is to provide a document or tutorial of the current capabilities of the platform and seek input from the DM teams on priorities for further development.

      (Note this epic will start late as the epics behind it are not closed yet).

        Attachments

        Stories in Epic (Custom Issue Matrix)

        Key Summary Story Points Assignee Status
         
        		DM-12606

        Refactor bokeh apps for accessing data from the Flask REST API and from S3

        		 4.2 Angelo Fausti Done
         
        		DM-12604

        Migrate SQuaSH data from the current production database to the new database schema

        		 8 Adam Thornton Done
         
        		DM-12603

        Deployment of the squash-restful-api service

        		 2.8 Angelo Fausti Done
         
        		DM-12601

        Implement flasgger for the SQuaSH REST API documentation

        		 2.8 Angelo Fausti Done
         
        		DM-12194

        Redesign SQuaSH REST API to make it complaint with lsst.verify

        		 7 Angelo Fausti Done
         
        		DM-12191

        Definition of the Flask extensions to be used in the SQuaSH REST API

        		 4.2 Angelo Fausti Done
         
        		DM-13490

        Retrieve status of job creation from the API

        		 2.8 Angelo Fausti Done
         
        		DM-13446

        Redeploy squash-api to a n1-standard-4 node

        		 1.4 Angelo Fausti Done
         
        		DM-14254

        Miscellaneous fixes

        		 1.4 Angelo Fausti Done
         
        		DM-14185

        Set up a sandbox instance for SQuaSH

        		7 Angelo Fausti Done
         
        		DM-14180

        Fix nagios checks after new squash deployment

        		 0.2 Angelo Fausti Done
         
        		DM-14151

        Set up a new production instance and synchronize databases

        		 4.2 Angelo Fausti Done
         
        		DM-13414

        Create the /code_changes resource in the Flask REST API to feed the monitor app

        		 2.8 Angelo Fausti Done
         
        		DM-13398

        Tests of squash-restful-api with current test datasets used in CI

        		 2.8 Angelo Fausti Done
         
        		DM-13394

        Increase default http timeout in lsst.verify

        		 1.4 Angelo Fausti Done
         
        		DM-13994

        Add code changes back to the Monitor app

        		5.6 Angelo Fausti Done
         
        		DM-14277

        Fix dataset names in squash and add ability to ignore a particular dataset

        		 1.4 Angelo Fausti Done
         
        		DM-9242

        squash KPM plots should label the time axis with a timezone

        		 0.5 Angelo Fausti Done
         
        		DM-8836

        Implement versioning in SQuaSH REST API

        		 1.4 Angelo Fausti Done
         
        		DM-9741

        Implement a minimal viable REST API in Flask with a test client

        		7 Angelo Fausti Done
         
        		DM-13237

        Implement authentication via access token in lsst.verify

        		 1.4 Angelo Fausti Done
         
        		DM-13236

        Create the /monitor resource in the Flask REST API to feed the monitor app

        		 2.8 Angelo Fausti Done
         
        		DM-13174

        Redirect root URL to /apidocs

        		 0.2 Angelo Fausti Done
         
        		DM-12807

        Migrate SQuaSH database to Google Cloud SQL

        		 1.4 Angelo Fausti Done
         
        		DM-12964

        Implement tests for the SQuaSH REST API

        		 2.8 Angelo Fausti Done
         
        		DM-13275

        Upload verification job and data blobs to an S3 bucket using Celery

        		 4.2 Angelo Fausti Done

          Activity

          Ascending order - Click to sort in descending order
          Hide
          Permalink
          afausti Angelo Fausti added a comment -

          Moving the stories related to the API migration from Django to Flask into this epic.

          Show
          afausti Angelo Fausti added a comment - Moving the stories related to the API migration from Django to Flask into this epic.
          Hide
          Permalink
          afausti Angelo Fausti added a comment -

          As a result of this epic, we have a new implementation fo the SQuaSH RESTful API in Flask, compatible with lsst.verify and uploading data blobs to AWS S3. It is deployed at https://squash.lsst.codes and have been collecting data from CI runs for ~2 weeks now.

          SQR019 was updated to demonstrate its use with lsst.verify and a sandbox instance of SQuaSH was deployed as requested by the System Verification group.

          Show
          afausti Angelo Fausti added a comment - As a result of this epic, we have a new implementation fo the SQuaSH RESTful API in Flask, compatible with lsst.verify and uploading data blobs to AWS S3. It is deployed at https://squash.lsst.codes and have been collecting data from CI runs for ~2 weeks now. SQR019 was updated to demonstrate its use with lsst.verify and a sandbox instance of SQuaSH was deployed as requested by the System Verification group.
          Hide
          Permalink
          frossie Frossie Economou added a comment -

          The DJANGO to flask migration is done in in production.

          Further input will be provided via the QA working group. 

          Show
          frossie Frossie Economou added a comment - The DJANGO to flask migration is done in in production. Further input will be provided via the QA working group. 

            People

            Assignee:
            afausti Angelo Fausti
            Reporter:
            frossie Frossie Economou
            Reviewers:
            Frossie Economou
            Watchers:
            Angelo Fausti, Frossie Economou
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                Jenkins

                No builds found.