To simplify dependencies of daf_butler and pipe_base on C++ packages we propose:
- Move the C++ code out of utils and into a cpputils package.
- Have utils depend on cpputils from an EUPS perspective for backwards compatibility during transition but have no code in utils depending on cpputils. utils itself will not depend on any other lsst packages (not base or pex_exceptions).
- Write python version of getPackageDir to be retained in utils package (only afw uses the C++ version)
- Move the utility code out of daf_butler and pipe_base into utils. Leave forwarding and deprecation messages behind in pipe_base – the lsst.daf.butler.core module is not meant to be public anyhow so no forwarding/deprecation is required.
- Change license of utils to BSD three clause.
- Move ButlerURI to entirely new (BSD3) package. Name not yet certain but likely to be butleruri, nor whether it has an "lsst." module prefix (astro_metadata_translator does not).
daf_butler and pipe_base include general purpose utility code that would benefit from being factored out into separate packages. daf_butler also uses one routine from utils (it was originally in daf_butler).
utils is not a good dependency fit for pipe_base or daf_butler because it has C++ code and therefore also brings in pex_exceptions and base. Splitting utils into a C++ package and a python package would help with this but more specific packages would be better.
- Put logging into log_utils. Now that Task has a special LoggerAdapter it would be nice if anyone could have access to .verbose and .trace without needing pipe_base. daf_butler currently defines its own VERBOSE constant. There are also the timer log utilities in pipe_base and separate ones in daf_butler.
- test_utils – there are a number of test routines in utils and daf_butler that could be combined.
Should these be imported as lsst.testutils and lsst.logutils – there are issues with namespace clashes for lsst.log.utils and lsst.utils.test.
There has also been a proposal for a pybind11_utils package from Jim Bosch.
The doImport routine in utils also needs a place outside of utils. I would welcome suggestions for a name.
If there is resistance to multiple small packages being created, one option is to have a new python-only utils package (pyutils ? Although import lsst.pyutils seems redundant) containing the python-only code from utils and the utility code from butler and pipe_base.
Additionally, I would like to extract the general purpose ButlerURI code from daf_butler and put it in its own package. This code forms the basis of all the datastore abstractions in butler but is a general purpose class for reading from and writing to object stores and local files without the caller code caring about the details (it supports WebDAV, S3, local files, pkg_resources). There are places here it would be convenient to be able to use this functionality without having to depend on daf_butler. By default I would call the package butleruri – from lsst.butleruri import ButlerURI. It should be standalone (not part of another utils package) and I intend to put it on PyPI. I would also like permission to relicense it as BSD 3 clause.
Any functions that exist in a formal release would have deprecated wrappers added in the original location that would forward to the new name.