Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-26035

Parallelize lsst_build with asyncio

    Details

    • Type: Story
    • Status: Reviewed
    • Resolution: Unresolved
    • Fix Version/s: None
    • Component/s: lsst_build
    • Labels:
      None
    • Team:
      Architecture
    • Urgent?:
      No

      Description

      lsst_build should be parallelized using asyncio, particularly git checkouts and git lfs checkouts. If possible, some care should be taken so that tarball download, if implemented, could also be executed in parallel.

        Attachments

          Issue Links

            Activity

            Hide
            bvan Brian Van Klaveren added a comment -

            A list of improvements which are largely ordered by commit:

            • New implementation of topological sort. The topological sort now can return lists-of-lists, where each list can be processed in parallel, in addition to each sub-list being sorted which provides determinism. We do not use this at this time, but it will be useful for tarball downloading and installation.
            • Creation of the models.py module, to move code out of prepare which might be needed by additional modules (prevent circular imports).
            • Move away from using smudge on checkout to using git-lfs pull after checking out git-lfs repos. This helps us build a manifest before downloading all git-lfs data.
            • VersionDbGit now relies solely on the hash of dependencies when calculating the version, instead of building a sequence number and relying on files in VersionDB, as per RFC-714.
              Improve versioning in setup.py. This adds scm-based versioning and relies on tags of the form lsst-build-v[VERSION] when calculating a version at build time. It also adds support for git-archive, which is particularly useful when building conda recipes. (For this to pass travis, I did have to push a dummy tag to a commit somewhere on the master branch, since it relies on git-describe when determining the version from git.)

             

            Show
            bvan Brian Van Klaveren added a comment - A list of improvements which are largely ordered by commit: New implementation of topological sort. The topological sort now can return lists-of-lists, where each list can be processed in parallel, in addition to each sub-list being sorted which provides determinism. We do not use this at this time, but it will be useful for tarball downloading and installation. Creation of the models.py module, to move code out of prepare which might be needed by additional modules (prevent circular imports). Move away from using smudge on checkout to using git-lfs pull after checking out git-lfs repos. This helps us build a manifest before downloading all git-lfs data. VersionDbGit now relies solely on the hash of dependencies when calculating the version, instead of building a sequence number and relying on files in VersionDB, as per RFC-714 . Improve versioning in setup.py. This adds scm-based versioning and relies on tags of the form lsst-build-v [VERSION] when calculating a version at build time. It also adds support for git-archive, which is particularly useful when building conda recipes. (For this to pass travis, I did have to push a dummy tag to a commit somewhere on the master branch, since it relies on git-describe when determining the version from git.)  
            Hide
            bvan Brian Van Klaveren added a comment -

            I've been using it this week without issue.

            Show
            bvan Brian Van Klaveren added a comment - I've been using it this week without issue.
            Hide
            rowen Russell Owen added a comment -

            Reviewed on github. I had a few minor suggestions to consider. Overall this looks like a very nice improvement.

            Show
            rowen Russell Owen added a comment - Reviewed on github. I had a few minor suggestions to consider. Overall this looks like a very nice improvement.

              People

              • Assignee:
                bvan Brian Van Klaveren
                Reporter:
                bvan Brian Van Klaveren
                Reviewers:
                Russell Owen, Tim Jenness
                Watchers:
                Brian Van Klaveren, Russell Owen, Tim Jenness
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:

                  Summary Panel