Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-11748

Clarify "Bulk Download" requirement

    Details

    • Team:
      Data Facility

      Description

      Requirement DMS-REQ-0300, "Bulk Download Service" would have much clearer force if it were reworded to something like:

      "The DMS shall make the released data products of the LSST available for bulk download, subject to the availability of network bandwidth and the application of data rights constraints. This shall include at least the current Data Release and all previous Data Releases from the operational phase of the Project."

      Otherwise, it is really not clear what is meant. What is this "software" that's mentioned in the current version? Something the user installs to provide them with access?

      Do we need to mention "raw data" explicitly? Perhaps we instead need to clarify somewhere that the raw image data is a formal "data product"?

      The nature of the service to be provided should begin to be constrained in the near future with an LDM-level document, as well. How fine-grained are the selections available? By DR? By table? By a subset of columns from a table? All images for my favorite 400 sq. deg. of sky? What network protocols / standards are involved?

      Note that the "all previous" stands in contrast to the OSS-REQ-0186-derived requirement (post-LCR-908) which envisions that the operations organization might choose to offer the normal ("non-bulk") data access services only to a subset of Data Releases.

      For reference, the current text is:

      The DMS shall provide software to enable bulk download of data products and raw data, subject to network bandwidth.

        Attachments

          Activity

          Hide
          ktl Kian-Tat Lim added a comment -

          If Bulk Download is only for released data products, then Level 1 data products are not included since they're never released. Furthermore, since the intended purpose of Level 1 data products is to enable rapid follow-up, it's a little hard to see how bulk download is applicable to them.

          Show
          ktl Kian-Tat Lim added a comment - If Bulk Download is only for released data products, then Level 1 data products are not included since they're never released. Furthermore, since the intended purpose of Level 1 data products is to enable rapid follow-up, it's a little hard to see how bulk download is applicable to them.
          Hide
          gpdf Gregory Dubois-Felsmann added a comment - - edited

          I added the word "released", compared to the previous text of DMS-REQ-0300, not to limit bulk download to "Data Releases" but to the data products made available to science users (as opposed to, say, intermediates). We do use the verb "release" in various places to describe making L1DPs available to science users, e.g., in OSS-REQ-0127 and OSS-REQ-0132, as well as in the SRD (e.g., in the phrase "continuously released").

          If our DPs weren't proprietary, I would have written "public data products" instead, but I try to avoid using that phrase for fear of causing a different type of confusion.

          Show
          gpdf Gregory Dubois-Felsmann added a comment - - edited I added the word "released", compared to the previous text of DMS-REQ-0300, not to limit bulk download to "Data Releases" but to the data products made available to science users (as opposed to, say, intermediates). We do use the verb "release" in various places to describe making L1DPs available to science users, e.g., in OSS-REQ-0127 and OSS-REQ-0132, as well as in the SRD (e.g., in the phrase "continuously released"). If our DPs weren't proprietary, I would have written "public data products" instead, but I try to avoid using that phrase for fear of causing a different type of confusion.
          Hide
          tjenness Tim Jenness added a comment -

          I think we've now decided that bulk download is owned by Robert Gruendl so I'm inclined to reassign this ticket to him such that we can decide whether it should be closed.

          Show
          tjenness Tim Jenness added a comment - I think we've now decided that bulk download is owned by Robert Gruendl so I'm inclined to reassign this ticket to him such that we can decide whether it should be closed.
          Hide
          gruendl Robert Gruendl added a comment -

          If there are no objections I will just reassign this to me.

           

          Show
          gruendl Robert Gruendl added a comment - If there are no objections I will just reassign this to me.  
          Hide
          tjenness Tim Jenness added a comment -

          I'm not objecting. Reassigned. Thanks.

          Show
          tjenness Tim Jenness added a comment - I'm not objecting. Reassigned. Thanks.

            People

            • Assignee:
              gruendl Robert Gruendl
              Reporter:
              gpdf Gregory Dubois-Felsmann
              Watchers:
              Gregory Dubois-Felsmann, Kian-Tat Lim, Leanne Guy, Michelle Butler, Robert Gruendl, Tim Jenness, Wil O'Mullane
            • Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

              • Created:
                Updated:

                Summary Panel