Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-31722

Move utility code out of pipe_base and daf_butler into utils package

    XMLWordPrintable

    Details

    • Story Points:
      8
    • Team:
      Architecture
    • Urgent?:
      No

      Description

      Move all the general purpose utility code out of pipe_base and daf_butler and into utils. The license for utils will be changed to BSD 3 clause and a setup.py will be created to allow it to be built and tested with a GitHub Action.

      daf_butler has no public APIs for this code and will be migrated to use utils with no deprecation period.

      pipe_base will include deprecation notices for code that existed in v22. The logging code that is new (and only used by pipe_base) will not include deprecation messages.

        Attachments

          Issue Links

            Activity

            Hide
            tjenness Tim Jenness added a comment -

            Jim Bosch there are a lot of pull requests here but all but utils are just making changes to keep up with the changes to utils, daf_butler, and pipe_base. The timer code has been made more general.

            utils now uses mypy (except for a couple of the functions in test.py and wrappers.py) so daf_butler is now able to use those when using mypy. Doing that forced me to fix doImport usage to do type validation and I also added doImportType to ensure that a type and not a module is retrieved. Note that you will need to set MYPYPATH to $UTILS_DIR/python otherwise things will not pass in daf_butler.

            I am deliberately not exporting these new lsst.utils functions into the default namespace.

            There are now only three routines left in butler utils.py: globToRegex, stripIfNotNone and transaction. Nothing uses those outside of daf_butler and the transaction one in particular seems quite specific. I could move the other two over if you think it's worth it, or else wait until something else needs them. I wasn't sure if you wanted the named.py stuff to move over or any of the time functions in daf_butler.

            Show
            tjenness Tim Jenness added a comment - Jim Bosch there are a lot of pull requests here but all but utils are just making changes to keep up with the changes to utils, daf_butler, and pipe_base. The timer code has been made more general. utils now uses mypy (except for a couple of the functions in test.py and wrappers.py) so daf_butler is now able to use those when using mypy. Doing that forced me to fix doImport usage to do type validation and I also added doImportType to ensure that a type and not a module is retrieved. Note that you will need to set MYPYPATH to $UTILS_DIR/python otherwise things will not pass in daf_butler. I am deliberately not exporting these new lsst.utils functions into the default namespace. There are now only three routines left in butler utils.py: globToRegex, stripIfNotNone and transaction. Nothing uses those outside of daf_butler and the transaction one in particular seems quite specific. I could move the other two over if you think it's worth it, or else wait until something else needs them. I wasn't sure if you wanted the named.py stuff to move over or any of the time functions in daf_butler.
            Hide
            jbosch Jim Bosch added a comment -

            Only minor comments, many of them preexisting.  Only even slightly big question is whether to rename iterable.

            Show
            jbosch Jim Bosch added a comment - Only minor comments, many of them preexisting.  Only even slightly big question is whether to rename iterable .
            Hide
            tjenness Tim Jenness added a comment -

            All branches merged. I did rename iterable to ensure_iterable.

            Show
            tjenness Tim Jenness added a comment - All branches merged. I did rename iterable to ensure_iterable .

              People

              Assignee:
              tjenness Tim Jenness
              Reporter:
              tjenness Tim Jenness
              Reviewers:
              Jim Bosch
              Watchers:
              Jim Bosch, Kian-Tat Lim, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.