Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-14185

Set up a sandbox instance for SQuaSH

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      The idea is to have an isolated environment where users can play with lsst.verify and squash.

      The sandbox will have its own CloudSQL db instance, and will run on a different k8s cluster where production is running.

      The instance will be deployed at https://squash-sandbox.lsst.codes

      • Simplify the monitor app removing dataset selection, code changes and drill down plots (at least for now)
      • Create a mechanism to define which bokeh apps are going to be initialized during deployment of the squash-bokeh and squash services
      • Re-work the SQuaSH initial page, the idea is to remove specific text, go straight to the main bokeh app and report statistics about the system on a secondary page
      • No changes in the squash-restful-api should be required
      • Some user documentation will be required, I plan to extend SQR-019 to include an example of dispatch_verify to the sandbox instance

        Attachments

          Activity

          Hide
          afausti Angelo Fausti added a comment - - edited

          Simplified Monitor App and making sure it doesn't break when no data is returned from the squash-restful-api. This is what users will see before sending any results to https://squash-sandbox.lsst.codes

          Show
          afausti Angelo Fausti added a comment - - edited Simplified Monitor App and making sure it doesn't break when no data is returned from the squash-restful-api. This is what users will see before sending any results to https://squash-sandbox.lsst.codes
          Hide
          afausti Angelo Fausti added a comment -

          See PR https://github.com/lsst-sqre/sqr-019/pull/2 for the SQR-019 technote update.

          Show
          afausti Angelo Fausti added a comment - See PR https://github.com/lsst-sqre/sqr-019/pull/2 for the SQR-019 technote update.
          Hide
          afausti Angelo Fausti added a comment - - edited

          Issues found along this implementation:

          1) On the squash-restful-api service, the celery worker complained about

           An error occurred (NoSuchBucket) when calling the PutObject operation: The specified bucket does not exist
          

          Added the step to create the S3 bucket in the deployment instructions. It is now suffixed by the NAMESPACE of the deployment, e.g.

          squash-sandbox.data

          2) Running dispatch_verify.py without the --env option caused this error in the SQUASH RESTful API:

          [2018-04-25 06:01:35,095] ERROR in job: Missing env metadata.
          

          Now if the environment information is missing SQUASH just creates an unknown environment and associates the job to it. This is needed for the sandbox instance or for any other use of SQUASH where the environment metadata is not required.

          3)

          verify.bin.dispatchverify.main INFO: Refreshing metric definitions from verify_metrics
          

          dispatch_verify.py always updates the Job with metrics definition from the verify_metrics package, this should be optional and disabled by default. For applications of SQUASH other than DM, the metrics and specification might be defined in external packages. There will be another PR for that.

          Show
          afausti Angelo Fausti added a comment - - edited Issues found along this implementation: 1) On the squash-restful-api service, the celery worker complained about An error occurred (NoSuchBucket) when calling the PutObject operation: The specified bucket does not exist Added the step to create the S3 bucket in the deployment instructions. It is now suffixed by the NAMESPACE of the deployment, e.g. squash-sandbox.data 2) Running dispatch_verify.py without the --env option caused this error in the SQUASH RESTful API: [2018-04-25 06:01:35,095] ERROR in job: Missing env metadata. Now if the environment information is missing SQUASH just creates an unknown environment and associates the job to it. This is needed for the sandbox instance or for any other use of SQUASH where the environment metadata is not required. 3) verify.bin.dispatchverify.main INFO: Refreshing metric definitions from verify_metrics dispatch_verify.py always updates the Job with metrics definition from the verify_metrics package, this should be optional and disabled by default. For applications of SQUASH other than DM, the metrics and specification might be defined in external packages. There will be another PR for that.
          Hide
          jsick Jonathan Sick added a comment - - edited

          The SQR-019 addition looks great.

          Show
          jsick Jonathan Sick added a comment - - edited The SQR-019 addition looks great.
          Show
          afausti Angelo Fausti added a comment - Other PRs associated with this ticket: https://github.com/lsst-sqre/squash-restful-api/pull/21 https://github.com/lsst-sqre/squash/pull/11 https://github.com/lsst-sqre/squash-bokeh/pull/13
          Hide
          afausti Angelo Fausti added a comment -

          Wrapping up last week work on the sandbox instance.

          Deployed changes to https://squash-demo.lsst.codes/ using production settings for testing before updating production.

          Show
          afausti Angelo Fausti added a comment - Wrapping up last week work on the sandbox instance. Deployed changes to https://squash-demo.lsst.codes/ using production settings for testing before updating production.

            People

            • Assignee:
              afausti Angelo Fausti
              Reporter:
              afausti Angelo Fausti
              Watchers:
              Angelo Fausti, Jonathan Sick, Michael Reuter, Simon Krughoff
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Summary Panel