Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-35047

Put middleware packages on pypi

    XMLWordPrintable

    Details

    • Team:
      Architecture
    • Urgent?:
      No

      Description

      Now that we have the ability to derive proper versions from the git repository, we need to update the packages to allow them to be properly distributed on PyPI.

      The relevant packages are:

      • sphgeom
      • pex_config
      • resources
      • utils
      • daf_butler
      • pipe_base
      • ctrl_mpexec
      • astro_metadata_translator

      I will leave ctrl_bps out for now.

      The process matches that in lsst-versions:

      • Choose names lsst-x (lsst-daf-butler, lsst-ctrl-mpexec etc).
      • Update the pyproject.toml and setup.cfg to work correctly following the PyPI specification.
      • Modify the GitHub action to upload to PyPI whenever a tag is pushed.

      Astro_metadata_translator tried to go it alone with versioning and tried to be semantic but it's filled with tags that look like v23.x and so at this point we probably should give up and make the next release align with all the other packages.

      I will add packages to PyPI incrementally so that packages can depend on the PyPI versions. Note that the PyPI sphgeom will use setuptools and so will not have a reusable C++ library. Python users will not notice but if we put anything on conda-forge we should probably use the cmake version.

      When we develop packages that depend on other packages we will have to temporarily edit the dependencies to refer to the github repo.

        Attachments

          Issue Links

          relates to

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-35191 Add test matrix for build workflow

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-35199 Fix python package names

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-35241 Add entry point for pipetask in ctrl_mpexec

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-35267 Fix race condition when uploading documentation

          • Done

            Activity

            Ascending order - Click to sort in descending order
            Hide
            Permalink
            tjenness Tim Jenness added a comment -

            Thanks for putting them on pypi. In the future we should try to see what other relevant classifiers we should use. Some comments:

            • Sphgeom
              • It lists "OS Independent" for the operating system but that can't be true.
              • The rendering of the readme is not quite right. Was the mime type listed correctly?
              • Aside: Looking at other spherical geometry packages I see STScI has one but the classifiers aren't helpful.
              • Aside: I wonder if I can do an RFC to change the license for sphgeom. It's such a core component.
            • Resources
              • No description text at all. You can convert the README.md to README.rst if that helps (although I thought pypi could take markdown)
              • License is BSD 3-clause not GPL.
              • Resources likely should not be listed as Astronomy topic. fsspec is similar but it doesn't help me. Intended audience of "Developers" seems like a good choice though.
            • utils
              • I clearly need to improve the README.
              • License is wrong. Should be BSD 3 not MIT.
              • Intended audience: developers I guess.
            • daf_butler
              • Seems okay.
            • pex_config
              • This is a dual license package. Can pypi support that? If it can't probably best to say BSD 3.
            • pipe_base
              • Ok.

            I can't find lsst-ctrl-mpexec.

            Show
            tjenness Tim Jenness added a comment - Thanks for putting them on pypi. In the future we should try to see what other relevant classifiers we should use. Some comments: Sphgeom It lists "OS Independent" for the operating system but that can't be true. The rendering of the readme is not quite right. Was the mime type listed correctly? Aside: Looking at other spherical geometry packages I see STScI has one but the classifiers aren't helpful. Aside: I wonder if I can do an RFC to change the license for sphgeom. It's such a core component. Resources No description text at all. You can convert the README.md to README.rst if that helps (although I thought pypi could take markdown) License is BSD 3-clause not GPL. Resources likely should not be listed as Astronomy topic. fsspec is similar but it doesn't help me. Intended audience of "Developers" seems like a good choice though. utils I clearly need to improve the README. License is wrong. Should be BSD 3 not MIT. Intended audience: developers I guess. daf_butler Seems okay. pex_config This is a dual license package. Can pypi support that? If it can't probably best to say BSD 3. pipe_base Ok. I can't find lsst-ctrl-mpexec.
            Hide
            Permalink
            wittgen Matthias Wittgen added a comment -

             

            Show
            wittgen Matthias Wittgen added a comment - pex_config dual licenses work in PyPi, fixed https://test.pypi.org/project/lsst-pex-config/ utils copied BSD LICENSE file, intended audience changed to Developers https://test.pypi.org/project/lsst-utils/ resources changed license pointed long description to README.md changed audience to Developers removed astronomy topic https://test.pypi.org/project/lsst-resources/  
            Hide
            Permalink
            wittgen Matthias Wittgen added a comment - - edited

            lsst-ctrl-mpexec is added: https://test.pypi.org/project/lsst-ctrl-mpexec/

            I am trying to build lsst-sphgeom for MacOS, Linux x arm, Intel with cibuildwheel

            Show
            wittgen Matthias Wittgen added a comment - - edited lsst-ctrl-mpexec is added: https://test.pypi.org/project/lsst-ctrl-mpexec/ I am trying to build lsst-sphgeom for MacOS, Linux x arm, Intel with cibuildwheel
            Hide
            Permalink
            wittgen Matthias Wittgen added a comment -
            Show
            wittgen Matthias Wittgen added a comment - sphgeom fixed OS list added MacOS build fixed README.md syntax https://test.pypi.org/project/lsst-sphgeom
            Hide
            Permalink
            wittgen Matthias Wittgen added a comment -

            Everything is merged, just closing out this ticket.

            Show
            wittgen Matthias Wittgen added a comment - Everything is merged, just closing out this ticket.
            Hide
            Permalink
            tjenness Tim Jenness added a comment -

            It's great. Thanks.

            Show
            tjenness Tim Jenness added a comment - It's great. Thanks.

              People

              Assignee:
              wittgen Matthias Wittgen
              Reporter:
              tjenness Tim Jenness
              Reviewers:
              Tim Jenness
              Watchers:
              Jim Bosch, Kian-Tat Lim, Matthias Wittgen, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.