Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-18883

Design a system for making example datasets accessible to pipelines.lsst.io documentation users and documentation CI


    • Type: Story
    • Status: To Do
    • Resolution: Unresolved
    • Fix Version/s: None
    • Component/s: None
    • Labels:


      As part of implementing QAWG-REC-15, we need datasets to base data processing examples and tutorials on in https://pipelines.lsst.io. Some of the requirements for these datasets might be:

      • Accessibility to the documentation CI infrastructure
      • Accessibility to end users, so that they can follow and reproduce the example (and assuming that these end users do not have access to the LSST Science Platform)
      • Versioning, so that datasets known to work with one version of the LSST Science Pipelines continue to be available as we roll out datasets for a newer version of the LSST Science Pipelines.

      The purpose of this ticket is to do research and propose, likely through a technote, a system for managing these example datasets. This ticket may even address the above requirements (for example, to state that examples should only be runnable from the LSST Science Platform).

      One option is to assume Git LFS, but this has significant real-world usability issues. Another option is a base HTTP server with a download client (as done by Astropy, for example). An S3 Butler backend could also work, once available.

      RFC-243 is yielding datasets that are accessible to internal users, which may be useful for documentation examples. Hopefully the documentation example dataset effort can piggy-back as much as possible on that.


          Container Issues



              • Assignee:
                jsick Jonathan Sick
                jsick Jonathan Sick
                John Swinbank, Jonathan Sick
              • Votes:
                0 Vote for this issue
                2 Start watching this issue


                • Created:

                  Summary Panel