Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-12606

Refactor bokeh apps for accessing data from the Flask REST API and from S3

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Once the Flask based API is deployed and jointcal data is flowing to SQuaSH we can finally enable jointcal visualization in the SQuaSH Monitor app.

        Attachments

          Issue Links

            Activity

            Hide
            afausti Angelo Fausti added a comment - - edited

            Here is how one can access the data blobs from the SQuaSH API.

            Blobs are stored on S3. If you retrieve a specific job you see the references to the s3_uri in there.

            Example, for job id 885:

            https://squash-restful-api-demo.lsst.codes/job/885

            Each metric measurement can have several data blobs associated to it.

            To retrieve a particular data blob one has to specify in addition to the job id, the metric name and the name of the data blob, like this:

            https://squash-restful-api-demo.lsst.codes/blob/885?metric=validate_drp.AM1&name=MatchedMultiVisitDataset

            the data access is optimized in the sense that we access the exact bit of data we need for a specific drill down plot.

             GET /blob/885?metric=validate_drp.AM1&name=MatchedMultiVisitDataset => generated 491737 bytes in 1118 msecs (HTTP/2.0 200) 2 headers in 75 bytes (1 switches on core 1)
            
            

            Show
            afausti Angelo Fausti added a comment - - edited Here is how one can access the data blobs from the SQuaSH API. Blobs are stored on S3. If you retrieve a specific job you see the references to the s3_uri in there. Example, for job id 885: https://squash-restful-api-demo.lsst.codes/job/885 Each metric measurement can have several data blobs associated to it. To retrieve a particular data blob one has to specify in addition to the job id, the metric name and the name of the data blob, like this: https://squash-restful-api-demo.lsst.codes/blob/885?metric=validate_drp.AM1&name=MatchedMultiVisitDataset the data access is optimized in the sense that we access the exact bit of data we need for a specific drill down plot. GET /blob/885?metric=validate_drp.AM1&name=MatchedMultiVisitDataset => generated 491737 bytes in 1118 msecs (HTTP/2.0 200) 2 headers in 75 bytes (1 switches on core 1)
            Hide
            afausti Angelo Fausti added a comment - - edited

            It's interesting that the "acceptable" size for data blobs retrieved from the squash api ~<10k data points is comparable to the maximum size of data a bokeh plot can handle - and that's not coincidence, it is just the network/browser limitation.

            The solution for larger datasets would involve running datashader on the server side and transfer through http only the rasterized images for a given pair of coordinates and perhaps other parameters representing the zoom and view limits, if it works that would be really awesome.

            Show
            afausti Angelo Fausti added a comment - - edited It's interesting that the "acceptable" size for data blobs retrieved from the squash api ~<10k data points is comparable to the maximum size of data a bokeh plot can handle - and that's not coincidence, it is just the network/browser limitation. The solution for larger datasets would involve running datashader on the server side and transfer through http only the rasterized images for a given pair of coordinates and perhaps other parameters representing the zoom and view limits, if it works that would be really awesome.
            Show
            afausti Angelo Fausti added a comment - See PRs: https://github.com/lsst-sqre/squash/pull/9 https://github.com/lsst-sqre/squash-restful-api/pull/19 https://github.com/lsst-sqre/squash-bokeh/pull/11
            Hide
            afausti Angelo Fausti added a comment -

            Current master deployed to https://squash-demo.lsst.codes/

            Show
            afausti Angelo Fausti added a comment - Current master deployed to https://squash-demo.lsst.codes/

              People

              • Assignee:
                afausti Angelo Fausti
                Reporter:
                afausti Angelo Fausti
                Watchers:
                Angelo Fausti
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel