Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-16251

Dask Investigations for Notebooks and Scientific Input for SQuaRE Science Platform services

    Details

    • Type: Epic
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: JupyterLab
    • Labels:
    • Epic Name:
      sqre-f18-sci-2
    • Story Points:
      35
    • WBS:
      1.02C.10.02
    • Team:
      SQuaRE
    • Cycle:
      Fall 2018

      Description

      This epic covers scientific input into SQuaRE Science Platform services including the  development of example notebooks appropriate to Commissioning and other early users,and science-level design input on aspects of SQuaRE developed services.

      In this cycle we are demonstrating the use of dask for next-to-notebook processing and continuing to support commissioning's use of DM systems. 

       

        Attachments

          Issue Links

          Stories in Epic (Custom Issue Matrix)

            Activity

            Hide
            frossie Frossie Economou added a comment -

            We investigated use of dask in notebooks as a way of providing scalable catalogue operations in a pythonic manner using parquet files. We  created challenging notebooks such as one that draws a source density map of all 1.7 billion rows in the Gaia DR2 catalogue (see https://github.com/lsst-sqre/notebook-demo/tree/master/experiments/DASK-notebooks ). These notebooks were used to demonstrate the capabilities (and drive usecases) of the infrastructure work in DM-15718

            Show
            frossie Frossie Economou added a comment - We investigated use of dask in notebooks as a way of providing scalable catalogue operations in a pythonic manner using parquet files. We  created challenging notebooks such as one that draws a source density map of all 1.7 billion rows in the Gaia DR2 catalogue (see https://github.com/lsst-sqre/notebook-demo/tree/master/experiments/DASK-notebooks  ). These notebooks were used to demonstrate the capabilities (and drive usecases) of the infrastructure work in DM-15718 . 

              People

              • Assignee:
                krughoff Simon Krughoff
                Reporter:
                frossie Frossie Economou
                Watchers:
                Frossie Economou
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel