Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-14873

Clarify what the batch and other parallel processing capabilities exposed to users in the Science Platform will be

    Details

    • Epic Name:
      sst-User-Batch-Parallel-LSP
    • Team:
      DM Science

      Description

      We need to decide on (and add to the Science Platform Design document, LDM-542) a description of what the batch processing and other parallel computing capabilities available to science users in the Science Platform will be, and how they will be accessed. Presumably that access would be primarily from the Notebook Aspect, but this may also be related to the final form that emerges for the "next-to-database processing" system provided by the API Aspect.

      Under "other parallel computing" we should be including capabilities like providing Dask, or something comparable, to users.

      It is important to ensure that a non-staff perspective is maintained in the design, even though for the next few years the use of these capabilities will be dominated by staff.

      I've written this as an Epic because I can imagine a series of stories emerging as this is fleshed out: perhaps initially writing a set of use cases, then a tech note with a possible design, then a formal addition to LDM-542.

      This work needs to be completed in time for the external review of the Science Platform design.

        Attachments

          Issue Links

            Activity

            Hide
            lguy Leanne Guy added a comment -

            This should be clarified for the LSP review; 2019-04-10

            Show
            lguy Leanne Guy added a comment - This should be clarified for the LSP review; 2019-04-10
            Hide
            lguy Leanne Guy added a comment -

            Colin Slater 's technote DMTN-086 addresses the use cases. 

            Show
            lguy Leanne Guy added a comment - Colin Slater 's technote DMTN-086 addresses the use cases. 
            Hide
            lguy Leanne Guy added a comment -

            Work was done on this for the LSP review where batch and other parallel processing capabilities were presented.  LSE-319 was updated to describe what is meant by "next-to-data" processing. 

            Gregory Dubois-Felsmann were the necessary changes propagated to  LDM-542? Is this work complete?

            Show
            lguy Leanne Guy added a comment - Work was done on this for the LSP review where batch and other parallel processing capabilities were presented.  LSE-319 was updated to describe what is meant by "next-to-data" processing.  Gregory Dubois-Felsmann were the necessary changes propagated to  LDM-542? Is this work complete?
            Hide
            gpdf Gregory Dubois-Felsmann added a comment -

            Nothing was done about putting this in LDM-542 at the time, because although we have a "vision"-level plan, there were still quite a few questions about implementation.

            I don't think that the "batch" situation has really been clarified yet, even now. Frossie Economou wants to think about this a bit, I believe.

            I will consult with Frossie and report next week.

            Show
            gpdf Gregory Dubois-Felsmann added a comment - Nothing was done about putting this in LDM-542 at the time, because although we have a "vision"-level plan, there were still quite a few questions about implementation. I don't think that the "batch" situation has really been clarified yet, even now. Frossie Economou wants to think about this a bit, I believe. I will consult with Frossie and report next week.

              People

              • Assignee:
                gpdf Gregory Dubois-Felsmann
                Reporter:
                gpdf Gregory Dubois-Felsmann
                Watchers:
                Colin Slater, Frossie Economou, Gregory Dubois-Felsmann, Leanne Guy, Michael Wood-Vasey, Robert Gruendl, Simon Krughoff
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Due:
                  Created:
                  Updated:

                  Summary Panel