Uploaded image for project: 'Request For Comments'
  1. Request For Comments
  2. RFC-966

New Git LFS service (read if you maintain a Git LFS hosted repo)

    XMLWordPrintable

Details

    • RFC
    • Status: Adopted
    • Resolution: Unresolved
    • DM, LSST
    • None

    Description

      Context:

      DM (but also related subsystems) use Git LFS to efficiently store large binary files into Git. To enable this, SQuaRE operates a Git LFS service that hosts the files. 

      The current service is a maintainability headache because

      (a) it is based on a third-party package that is abandonware (archived in 2018)

      (b) It is written in ruby, and so we have no interest in taking it over

      (c) it is hosted on AWS (instead of our current cloud platform contractor, GCP)

      (d) it predates our current deployment infrastructure (aka it's a snowflake)

      Github does provide a hosted Git LFS service but

      (a) it is expensive and more importantly

      (b) it has a file limit size of 5GB and on consultation, there are folks planning to store files that exceed this limit. 

       

      Proposal being RFCd:

      We have implemented a new service, based on our current service deployment infrastructure (phalanx). This is based on a different third-party package, giftless https://github.com/datopian/giftless which has been active more recently and is implemented in python.

      We propose to retire the current Git LFS service and replace it with this new Git LFS service. 

      Impact:

      1. Current repos need to be migrated to the new service. We are willing to do this work, or explain to repo owners how to do it if they prefer. 

      2. For technical reasons, this change will not be transparent to all users, as it requires special handling to obtain write (push) access to Git LFS repository. To help you understand this impact, we have written the "what if" change to the developer guide that contains the instructions as they would be for this new service. 

      If you prefer text you can compare 

      https://developer.lsst.io/git/git-lfs.html# (current instructions) with
      https://developer.lsst.io/v/DM-40799/git/git-lfs.html (proposed new instructions)

      Or you can look at the diff:

      https://github.com/lsst-dm/dm_dev_guide/pull/638/files#diff-bb993c86fc07e76a61232a61050239652fbdff1222b7752366b180c6534e6582

       

      The summary is that if you are a repo maintainer and need to push to a Git LFS repo under the proposed change, you will have to use a different URL in your configuration and use a different process to cache your credentials. 

      Discussion:

      While this creates a slight complication for current maintainers of Git LFS repo, we have no other choice if we wish to address the technical debt of running an obsolete service. If this RFC cannot gain consensus to pass, the alternative would be to pay for Github hosted service (and live with the 5GB size limit) or find a volunteer wishing to support the current service. 

       

       

       

       

      Attachments

        Issue Links

          Activity

            It's in addition to (and for a different URL).  You'll still need the https push token for GitHub to push all the non-LFS stuff.

            It is necessary to edit the repositories because the read and write URLs are not the same.  If you do not need to push the large binary artifacts you do not need to do anything.  If you do need to push the large files in Git LFS, we encourage you to be careful.

            athornton Adam Thornton added a comment - It's in addition to (and for a different URL).  You'll still need the https push token for GitHub to push all the non-LFS stuff. It is necessary to edit the repositories because the read and write URLs are not the same.  If you do not need to push the large binary artifacts you do not need to do anything.  If you do need to push the large files in Git LFS, we encourage you to be careful.

            Yes, but why are there different read and write URLs? That's not exactly the standard way to use LFS.

            krzys Krzysztof Findeisen added a comment - Yes, but why are there different read and write URLs? That's not exactly the standard way to use LFS.
            rra Russ Allbery added a comment -

            There are different read and write URLs because unifying them requires us to build and maintain a custom server container with a modified authentication flow plugin in order to implement conditional authentication based on the operation being performed. With different read and read/write URLs, we can move all permission checking to the Kubernetes ingress using the tools we developed for the RSP, and can use the upstream code as-is.

            rra Russ Allbery added a comment - There are different read and write URLs because unifying them requires us to build and maintain a custom server container with a modified authentication flow plugin in order to implement conditional authentication based on the operation being performed. With different read and read/write URLs, we can move all permission checking to the Kubernetes ingress using the tools we developed for the RSP, and can use the upstream code as-is.
            ktl Kian-Tat Lim added a comment - - edited

            Note that the "manual editing" is minimally error-prone. Since you can do the "editing" (the git config commands, and really only one of them) at any time, the absolute worst case is that the push fails, you execute the command, and things then work. If you get the URL wrong in the "edit", it fails and you fix it.

            ktl Kian-Tat Lim added a comment - - edited Note that the "manual editing" is minimally error-prone. Since you can do the "editing" (the git config commands, and really only one of them) at any time, the absolute worst case is that the push fails, you execute the command, and things then work. If you get the URL wrong in the "edit", it fails and you fix it.

            I think we are okay with this, so I am going to go ahead and adopt and we will proceed with the implementation

            frossie Frossie Economou added a comment - I think we are okay with this, so I am going to go ahead and adopt and we will proceed with the implementation

            People

              frossie Frossie Economou
              frossie Frossie Economou
              Adam Thornton, Christopher Waters, Eric Bellm, Frossie Economou, Ian Sullivan, John Parejko, Kian-Tat Lim, Krzysztof Findeisen, Lynne Jones, Russ Allbery, Yusra AlSayyad
              Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

                Created:
                Updated:
                Planned End:

                Jenkins

                  No builds found.