Uploaded image for project: 'Request For Comments'
  1. Request For Comments
  2. RFC-104

GitLFS as a solution to storing data files in repos


    • Type: RFC
    • Status: Implemented
    • Resolution: Done
    • Component/s: DM
    • Labels:
    • Location:
      This ticket or SQuaRE hipchat room


      SQuaRE proposes to use the open source GitLFS protocol as a solution to storing large files in repos in general, and binary-heavy repos like awfdata in particular. We are proposing this solution after evaluation because it has a workflow very close to what people are used to already, and it allows almost normal interaction with GitHub, while storing the files on an in-house backend server so that we do not have to pay hosting fees for the data. We are happy that this is a maintainable solution and friendly to non-project contributors; while GitLFS is still a young project barely out of beta, we have noted a good rate of bugfixing and have had good interactions with the development team. It also has had good adoption beyond GitHub, from both Microsoft and Atlassian.

      As part of the evaluation, J Matt Peterson [X] deployed a GitLFS cloud-based server on the NCSA Nebula OpenStack cluster which we now invite you to try out. We have set up a test repository in the lsst organisation that is seeded with afwdata:


      Feel free to do whatever you want with this, it will be deleted once the RFC is over. If you are in the Data Management team, you should be able to push to the repo.

      Instructions on how to interact with the repository are in the README:


      Note that this is a LARGE repository (intentionally) so expect some initial setup time and make sure you're on good bandwidth.

      Feel free to improve the README for particular platforms, etc. We will save it for inclusion in the real afwdata if the RFC is adopted.


          Issue Links



              • Assignee:
                frossie Frossie Economou
                frossie Frossie Economou
                Frossie Economou, J Matt Peterson [X] (Inactive), John Swinbank, Jonathan Sick, Joshua Hoblitt, Kian-Tat Lim, Tim Jenness
              • Votes:
                0 Vote for this issue
                7 Start watching this issue


                • Created:
                  Planned End:

                  Summary Panel