Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-32408

Write Python code to determine version string for pipeline package

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: lsst_versions, utils
    • Labels:
      None
    • Story Points:
      10
    • Team:
      Architecture
    • Urgent?:
      No

      Description

      Now that we are developing pip installable python packages (daf_butler, utils, sphgeom) and also supporting cmake (DM-23308), determining the version number to be consistent with the version calculated by EUPS with pkgautoversion is difficult.

      To distribute a package on PyPI we need a version number that is compatible with Python version standards (PEP-440) and pkgautoversion does not do that. Russ Allbery has proposed that we use a combination of the formal release and the weekly.

      For example in daf_butler at the moment we have:

      6f853fa8 - (HEAD -> master, tag: w.2021.44)

      Since we have created a tag v23.0.0.rc1 on w.2021.40 this commit is going to be part of v24. The suggestion is that we should call it 24.0.0a2021.44 – an alpha release for v24. EUPS calls this version 22.0.1-87-g6f853fa8 and it will continue to use 22.0.1 as the basis for it until 23.0.0 is released (but if 23.0.0 ends up being on a branch it won't ever pick it up).

      If it is assumed that there is always either a v23.0.0.rcx tag, a v23.0.x branch, or a 23.0.0 tag (if no backports were done) then it seems plausible for code to work out these alpha releases. It does though require that we don't suddenly change the release to be based on a new weekly since that would suddenly lead to the 24.0.0a.2021.44 becoming 23.0.0rc1.

      We can't use setuptools_scm directly because the tag scheme does not work. The proposal is to add some code to the utils package (noting that this might lead to base having to depend on utils if cmake is used for it) that cmake scripts can execute and which setup.py can use to determine a version. If it's in python we can't have it in lsst/utils/ since utils would not be able to import it without version.py existing (but we could make utils init.py clever enough that it works the version out itself if the version.py is not present). Should it be a shell script? Do we care that the version from pkgautoversion is going to be different?

        Attachments

          Issue Links

          mentioned in

          Page Loading...

            Activity

            Ascending order - Click to sort in descending order
            5 older comments
            Hide
            Permalink
            tjenness Tim Jenness added a comment -

            Kian-Tat Lim after a lot of struggling I seem to have something that works.

            • I looked at setuptools_scm and worked out how to add this code such that it is called automatically by setuptools when the package is built.
            • There is a chicken and egg problem in utils itself in that it needs to work out its own version number and can't work that out using the same approach as packages that use lsst-utils.
            • I do wonder if I should put this new code into lsst.utils_scm (but in the same repo). One problem I have is that setuptools tries to import the entry point and that requires that all the utils dependencies that are always imported (numpy, deprecated) are present in the little setuptools sandbox. I work around them by declaring numpy+deprecated as build system requirements but that's less than ideal.
            • I made a PR on resources to switch it to using utils to work out the version number to prove it worked. I think resources should get a v23.0.0rc1 tag so that it starts to report that it is part of v23.
            • Note that the version number calculated on the PR for utils is different to the one that you see for utils in the resources PR. This is because GitHub creates a merge commit of your branch and master so the number of commits going back to the most recent weekly differ.
            • I imagine we will have to support release version numbers as well so that if you run this on v24.0.1 tag it will calculate the proper answer and not a dev release. Can we save that for a future ticket?

            The commits will need some squashing because there has been a lot of fiddling with actions and setup.py to get things working.

            Show
            tjenness Tim Jenness added a comment - Kian-Tat Lim after a lot of struggling I seem to have something that works. I looked at setuptools_scm and worked out how to add this code such that it is called automatically by setuptools when the package is built. There is a chicken and egg problem in utils itself in that it needs to work out its own version number and can't work that out using the same approach as packages that use lsst-utils. I do wonder if I should put this new code into lsst.utils_scm (but in the same repo). One problem I have is that setuptools tries to import the entry point and that requires that all the utils dependencies that are always imported (numpy, deprecated) are present in the little setuptools sandbox. I work around them by declaring numpy+deprecated as build system requirements but that's less than ideal. I made a PR on resources to switch it to using utils to work out the version number to prove it worked. I think resources should get a v23.0.0rc1 tag so that it starts to report that it is part of v23. Note that the version number calculated on the PR for utils is different to the one that you see for utils in the resources PR. This is because GitHub creates a merge commit of your branch and master so the number of commits going back to the most recent weekly differ. I imagine we will have to support release version numbers as well so that if you run this on v24.0.1 tag it will calculate the proper answer and not a dev release. Can we save that for a future ticket? The commits will need some squashing because there has been a lot of fiddling with actions and setup.py to get things working.
            Hide
            Permalink
            tjenness Tim Jenness added a comment -

            As discussed, I will move the version code to a new package to remove the complication of dependencies and init files. I think something like lsst_version or lsst_scm.

            I will put this back into in progress.

            Show
            tjenness Tim Jenness added a comment - As discussed, I will move the version code to a new package to remove the complication of dependencies and init files. I think something like lsst_version or lsst_scm. I will put this back into in progress.
            Hide
            Permalink
            tjenness Tim Jenness added a comment -

            Kian-Tat Lim I've reorganized the code so that it's all now in lsst_versions and utils and resources have small pull requests that enable the version handling.

            Show
            tjenness Tim Jenness added a comment - Kian-Tat Lim I've reorganized the code so that it's all now in lsst_versions and utils and resources have small pull requests that enable the version handling.
            Hide
            Permalink
            ktl Kian-Tat Lim added a comment -

            A few suggestions but overall looks OK.

            Show
            ktl Kian-Tat Lim added a comment - A few suggestions but overall looks OK.
            Hide
            Permalink
            tjenness Tim Jenness added a comment -
            • lsst-versions is now on PyPI at https://pypi.org/project/lsst-versions/ and the GitHub repo is configured to push a new release every time a tag is added to the repo.
            • I modified all the middleware packages except ctrl_bps to use the versioning code (ctrl_bps is being reorganized).
            Show
            tjenness Tim Jenness added a comment - lsst-versions is now on PyPI at https://pypi.org/project/lsst-versions/ and the GitHub repo is configured to push a new release every time a tag is added to the repo. I modified all the middleware packages except ctrl_bps to use the versioning code (ctrl_bps is being reorganized).

              People

              Assignee:
              tjenness Tim Jenness
              Reporter:
              tjenness Tim Jenness
              Reviewers:
              Kian-Tat Lim
              Watchers:
              Fritz Mueller, Kian-Tat Lim, Russ Allbery, Tatiana Goldina, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.