Angelo Fausti added a comment - 11/Apr/18 3:46 PM - edited Here is how one can access the data blobs from the SQuaSH API. Blobs are stored on S3. If you retrieve a specific job you see the references to the s3_uri in there. Example, for job id 885: https://squash-restful-api-demo.lsst.codes/job/885 Each metric measurement can have several data blobs associated to it. To retrieve a particular data blob one has to specify in addition to the job id, the metric name and the name of the data blob, like this: https://squash-restful-api-demo.lsst.codes/blob/885?metric=validate_drp.AM1&name=MatchedMultiVisitDataset the data access is optimized in the sense that we access the exact bit of data we need for a specific drill down plot. GET /blob/885?metric=validate_drp.AM1&name=MatchedMultiVisitDataset => generated 491737 bytes in 1118 msecs (HTTP/2.0 200) 2 headers in 75 bytes (1 switches on core 1)

Angelo Fausti added a comment - 11/Apr/18 4:15 PM - edited It's interesting that the "acceptable" size for data blobs retrieved from the squash api ~<10k data points is comparable to the maximum size of data a bokeh plot can handle - and that's not coincidence, it is just the network/browser limitation. The solution for larger datasets would involve running datashader on the server side and transfer through http only the rasterized images for a given pair of coordinates and perhaps other parameters representing the zoom and view limits, if it works that would be really awesome.

Angelo Fausti added a comment - 17/Apr/18 1:29 PM See PRs: https://github.com/lsst-sqre/squash/pull/9 https://github.com/lsst-sqre/squash-restful-api/pull/19 https://github.com/lsst-sqre/squash-bokeh/pull/11

Angelo Fausti added a comment - 17/Apr/18 1:30 PM Current master deployed to https://squash-demo.lsst.codes/