Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-18505

Create SQuaSH execution environment for developers

    XMLWordPrintable

Details

    • Story
    • Status: Invalid
    • Resolution: Done
    • None
    • squash, verify
    • None

    Description

      SQuaSH and dispatch_verify.py support either jenkins or ldf as execution environments. To allow developers to send data to their sandbox instances, we need another execution environment. I would call it local.

      Also, review the policy whether we should allow the execution environment to be set as unkwon.

      Error message that motivated this ticket:

      [2019-03-15 17:14:41,757: ERROR/ForkPoolWorker-2] Task app.tasks.influxdb.job_to_influxdb[9380a5d0-3e79-4549-ae14-dcd2acfec892] raised unexpected: KeyError('env',)
      Traceback (most recent call last):
        File "/usr/local/lib/python3.6/site-packages/celery/app/trace.py", line 382, in trace_task
          R = retval = fun(*args, **kwargs)
        File "/usr/local/lib/python3.6/site-packages/celery/app/trace.py", line 641, in __protected_call__
          return self.run(*args, **kwargs)
        File "/opt/squash/app/tasks/influxdb.py", line 131, in job_to_influxdb
          if 'ci_dataset' in data['meta']['env']:
      KeyError: 'env'
      [2019-03-15 17:14:41,980: INFO/ForkPoolWorker-3] Task app.tasks.s3.upload_object[4dd045d3-d8f5-4863-bffa-459787f578f9] succeeded in 0.3280502166599035s: 's3://squash-sandbox.data/59'
      

      Attachments

        Issue Links

          Activity

            The metadata for the user local execution environment should include at least the user name, as authenticated through the SQuaSH API.

            afausti Angelo Fausti added a comment - The metadata for the user local execution environment should include at least the user name, as authenticated through the SQuaSH API.
            afausti Angelo Fausti added a comment - - edited

            The SQuaSH API should provide a generic resource to interact with verification jobs /jobs/<job_id> and specific resources to interact with runs within different execution environments that ultimately map to a verification job.

            A request to /local/<run_id>?user=<user name> will look up for the corresponding /jobs/<job_id> to retrieve the associated metric values and metadata.

            afausti Angelo Fausti added a comment - - edited The SQuaSH API should provide a generic resource to interact with verification jobs /jobs/<job_id> and specific resources to interact with runs within different execution environments that ultimately map to a verification job. A request to /local/<run_id>?user=<user name> will look up for the corresponding /jobs/<job_id> to retrieve the associated metric values and metadata.

            When this merges, the demo notebook LSSTVerificationSQuaSHDemo.ipynb should be updated to show how to use local execution environments.

            krughoff Simon Krughoff (Inactive) added a comment - When this merges, the demo notebook LSSTVerificationSQuaSHDemo.ipynb should be updated to show how to use local execution environments.
            tjenness Tim Jenness added a comment -

            afausti is this still an active ticket?

            tjenness Tim Jenness added a comment - afausti is this still an active ticket?

            Superceded by Saquatch APIĀ https://sasquatch.lsst.io/apis.html

            frossie Frossie Economou added a comment - Superceded by Saquatch APIĀ  https://sasquatch.lsst.io/apis.html

            People

              afausti Angelo Fausti
              afausti Angelo Fausti
              Angelo Fausti, Frossie Economou, Krzysztof Findeisen, Michael Reuter, Simon Krughoff (Inactive), Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.