Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-9017

Switch ndarray from a tarball to checked out source

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Won't Fix
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: ndarray

      Description

      We would like to switch ndarray from a tarball to a checked out fork of the github repo.

      So far I have:

      • Forked ndarray to lsst/ndarray-1 (since ndarray is the tarball version)
      • Created a branch named lsst-dev
      • Added a ups directory
      • Verified that it builds and tests pass on my machine

      (Note: Jim Bosch is willing to have the ups directory be part of the main repo, so we may not actually need an lsst-dev branch).

      I'm not sure of the next steps.

        Attachments

          Issue Links

            Activity

            Hide
            tjenness Tim Jenness added a comment - - edited

            If we are doing the "fork/branch" thing the trick is having minimal local changes. The more differences you have with upstream the harder it is to keep up to date. John Swinbank is right that we clearly failed at this last time ndarray was forked and I'm not sure why we ended up with such a divergent codebase that time. One option is to have a fork and agree never to do anything other than move where master points to. We can do that if upstream is happy to have a ups directory.

            Show
            tjenness Tim Jenness added a comment - - edited If we are doing the "fork/branch" thing the trick is having minimal local changes. The more differences you have with upstream the harder it is to keep up to date. John Swinbank is right that we clearly failed at this last time ndarray was forked and I'm not sure why we ended up with such a divergent codebase that time. One option is to have a fork and agree never to do anything other than move where master points to. We can do that if upstream is happy to have a ups directory.
            Hide
            jbosch Jim Bosch added a comment -

            Either way, I hope we are planning to keep using the upstream CMake build system rather than putting the Scons stuff back?

            Absolutely!

            Just wanted to mention a third option: do a `git clone` (of a specific upstream version) in the `eups fetch` stage instead of directly tracking master.
            That way there won't be a fork to diverge, and there will also not be a need to make a release plus tarball for every change.

            I don't have a technical argument against this, but I'd rather not add a new way to packaging things without a more compelling reason.

            One option is to have a fork and agree never to do anything other than move where master points to. We can do that if upstream is happy to have a ups directory.

            Upstream is happy to have a ups directory, and I strongly support not doing anything other than moving where master points to.

            Show
            jbosch Jim Bosch added a comment - Either way, I hope we are planning to keep using the upstream CMake build system rather than putting the Scons stuff back? Absolutely! Just wanted to mention a third option: do a `git clone` (of a specific upstream version) in the `eups fetch` stage instead of directly tracking master. That way there won't be a fork to diverge, and there will also not be a need to make a release plus tarball for every change. I don't have a technical argument against this, but I'd rather not add a new way to packaging things without a more compelling reason. One option is to have a fork and agree never to do anything other than move where master points to. We can do that if upstream is happy to have a ups directory. Upstream is happy to have a ups directory, and I strongly support not doing anything other than moving where master points to.
            Hide
            jbosch Jim Bosch added a comment -

            Wrote this earlier and just realized I forgot to actually post it, in response to John Swinbank's concerns about forks getting out of sync:

            I think we're basically just not throwing up the artificial obstacles we threw up last time.

            The old fork was more than just a fork: we created a new version of ndarray that used LSST's sconsUtils instead of the upstream build system and even moved all of its headers into LSST subdirectories and namespaces (which we later reverted, before we actually switched back to upstream). Those differences made it much more difficult to move changes between the forks, and by the time we reverted the namespace/subdirectory changes, things had diverged enough that putting them back together was a big pain.

            But unlike the other packages Tim Jenness mentioned, I also don't want us to make any changes on the master branch of the LSST fork that aren't already upstream; we can use LSST fork ticket branches to do work, but I always want those changes upstream before we start using them.

            Show
            jbosch Jim Bosch added a comment - Wrote this earlier and just realized I forgot to actually post it, in response to John Swinbank 's concerns about forks getting out of sync: I think we're basically just not throwing up the artificial obstacles we threw up last time. The old fork was more than just a fork: we created a new version of ndarray that used LSST's sconsUtils instead of the upstream build system and even moved all of its headers into LSST subdirectories and namespaces (which we later reverted, before we actually switched back to upstream). Those differences made it much more difficult to move changes between the forks, and by the time we reverted the namespace/subdirectory changes, things had diverged enough that putting them back together was a big pain. But unlike the other packages Tim Jenness mentioned, I also don't want us to make any changes on the master branch of the LSST fork that aren't already upstream; we can use LSST fork ticket branches to do work, but I always want those changes upstream before we start using them.
            Hide
            jbosch Jim Bosch added a comment - - edited

            I've been thinking about this, and I think the tricky part is making sure previous LSST-tagged versions of ndarray remain installable, since those will all be pulling from the same git repo. I think the following procedure should work, but I wanted to run it by all the interested parties to see if anyone can poke a hole in it:

            1. Pull everything in the current lsst/ndarray repo into a local clone, including tags.
            2. Rename master in the local clone to something new (maybe lsst-legacy).
            3. Delete the lsst/ndarray repo and replace it with a true GitHub fork of ndarray/ndarray.
            4. Push everything from the local clone back to lsst/ndarray.

            This should keep all of the commits that might be used by released LSST packages in the lsst/ndarray repo, while letting master still track upstream in the natural GitHub way. I'm not entirely certain how e.g. weekly releases that don't have git tags know what to checkout, but I think it must be using the SHA1 somehow, and that should still work with the above scheme.

            Show
            jbosch Jim Bosch added a comment - - edited I've been thinking about this, and I think the tricky part is making sure previous LSST-tagged versions of ndarray remain installable, since those will all be pulling from the same git repo. I think the following procedure should work, but I wanted to run it by all the interested parties to see if anyone can poke a hole in it: 1. Pull everything in the current lsst/ndarray repo into a local clone, including tags. 2. Rename master in the local clone to something new (maybe lsst-legacy ). 3. Delete the lsst/ndarray repo and replace it with a true GitHub fork of ndarray/ndarray. 4. Push everything from the local clone back to lsst/ndarray. This should keep all of the commits that might be used by released LSST packages in the lsst/ndarray repo, while letting master still track upstream in the natural GitHub way. I'm not entirely certain how e.g. weekly releases that don't have git tags know what to checkout, but I think it must be using the SHA1 somehow, and that should still work with the above scheme.
            Hide
            tjenness Tim Jenness added a comment -

            No longer relevant since we are now using the conda-forge ndarray.

            Show
            tjenness Tim Jenness added a comment - No longer relevant since we are now using the conda-forge ndarray.

              People

              Assignee:
              jbosch Jim Bosch
              Reporter:
              rowen Russell Owen
              Watchers:
              Jim Bosch, John Swinbank, Pim Schellart [X] (Inactive), Russell Owen, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.