Tatiana Goldina is away, but I will take a look at this from our outside-reuse viewpoint. Superficially it is certainly going in the right direction!

Can you be more specific about what would be moved where (preferably in the form of a list)? utils is kind of a mixed bag, and I don't know if anybody knows about everything that's in there, and the references to utility code in the higher-level packages are likewise ambiguous.

I'm not sure a pybind11_utils would be worth the effort, given that our pybind11 support at the moment is 4 C++ headers that have not been expanded in several years.

This is the content that I am talking about specifically:

utils

• Python
• deprecated (deprecate_pybind11 and suppress_deprecations)
• doImport
• get_caller_name
• inheritDoc
• test support code
• C++ (pulls in base and pex_exceptions)
• getPackageDir
• demangle
• backtrace

daf_butler

lsst.daf.butler.core.ButlerURI

Abstracted access to file resources. Automatically understands what to do with file:, s3:, https: and also python pkg_resources.
Would be useful outside of butler and would be even more useful if it was on pypi and had BSD license.

lsst.daf.butler.core.utils

• safeMakeDir (no longer needed now that os.makedirs takes exist_ok parameter)
• iterable (turn argument into an iterable, protecting str)
• allSlots (combines all slots in all parents)
• getFullTypeName
• getClassOf
• getInstanceOf
• Singleton (force class to be a singleton)
• transactional (decorator to call `with self.transaction`) – may be too specific
• stripIfNotNone (for stripping strings that might be None)
• immutable (decorator for class)
• cached_getter
• globToRegex
• isplit (str.split but using iterator to minimize memory footprint)

lsst.daf.butler.core.config

Dict-like class that supports hierarchical keys. Supports yaml and json serialization, and merging of configs.

lsst.daf.butler.core.time_utils

Code for converting to and from Astropy Time to integer nanoseconds and also supporting serialization of Astropy Time to YAML.

lsst.daf.butler.core.progress

Support for adding progress bars to command line applications.

lsst.daf.butler.core.logging

Logging support for Butler commands.

• VERBOSE log level
• ButlerMDC (allow additional content to be inserted into log records)
• ButlerLogRecord (a LogRecord-like class that accepts MDC)
• ButlerLogRecords (a collection of ButlerLogRecord)
• JsonLogFormatter (log formatter to write out JSON log records)
• ButlerLogRecordHandler (logging handler to accumulate ButlerLogRecord records)
• PrecisionLogFormatter (log formatter that reports timestamps in milliseconds)

pipe_base

lsst.pipe.base.task_logging

• Constants for TRACE and VERBOSE Logging
• TaskLogAdapter – logger for Task that supports .trace and .verbose
• `getLogger` equivalent that returns `TaskLogAdapter` rather than `Logger`

lsst.pipe.base.timer

The timer functions are completely generic and not Task-specific.

So one option is something like:

• Split out python from utils package to new package (name unknown: lsst.rubin.utils?).
• Have utils forward for a time for backwards compatibility.
• Move most of the code discussed above into that new pyutils package. (including lsst.rubin.utils.logging?)
• Pull ButlerURI into a brand new package (not sure what people would want to call that).
• Put lsst.rubin.utils onto pypi and allow BSD 3 clause license to allow it to be used by others.

Given discussions starting here it sounds like we should have a nice interface for easily configuring tasks instantiated in notebooks/standalone scripts to work again, as right now you need to do something along the lines of

import logging

logformat = "[%(levelname)s] %(name)s: %(message)s"

logging.basicConfig(format=logformat, level=logging.INFO)

before logging levels in the task and its subtasks are respected, and this is quite a mouthful to expect users to remember.

I dislike the name rubin_utils, especially after an initial lsst., as it seems to be claiming greater scope than is warranted.

I suggest that it might be possible to move the C++ code in the lsst.utils package out into a new package, maybe lsst.cpputils, reserving lsst.utils for Python-only utilities. Some C++ code like Demangle and Backtrace also might belong in base instead.

lsst.butleruri is not great as a name for a package intended for uses outside the Butler that is also about more than just URI manipulation but also retrieval. lsst.uridata might be better, but I understand that it might require significant code modification to implement.

I was not thinking of putting a `lsst` in front of the butleruri package name.

Looking at getPackageDir – the only place it is used in C++ is for some afw tests and some examples in order to find afw and afwdata. It might be worth moving that code to a private helper in afw and rewriting it in python so it could stay in utils.

utils currently has some C++ functions for unit conversion for magnitudes (jointcal uses them) and some pybind11 helpers.

I think we are ending up with the following proposal:

• Move the C++ code out of utils and into a cpputils package.
• Have utils depend on cpputils from an EUPS perspective for backwards compatibility during transition but have no code in utils depending on cpputils. utils itself will not depend on any other lsst packages (not base or pex_exceptions).
• Write python version of getPackageDir to be retained in utils package (only afw uses the C++ version)
• Move the utility code out of daf_butler and pipe_base into utils. Leave forwarding and deprecation messages behind in pipe_base – the lsst.daf.butler.core module is not meant to be public anyhow so no forwarding/deprecation is required.
• Change license of utils to BSD three clause.
• Move ButlerURI to entirely new (BSD3) package. Name not yet certain, nor whether it has an "lsst." module prefix (astro_metadata_translator does not).

We also need to talk about lsst.base.Packages – it is used by pipe_base and ctrl_mpexec for package provenance but because it's part of base it brings in unwanted compilers/lsst dependencies. Can it be moved to lsst.utils as part of this RFC or require a different RFC?