While I agree with all the statements above the only thing I'm not sure about is the timing. Are we already at the stage where we know all we want to configure and the ways in which we want it to be?
If yes, then great! Lets redesign. If not, then we might benefit from muddling on a tiny little bit longer until that becomes clear.

I think we need to do something about some of this now, even if we don't tackle all of it - we at least need a place to put default configuration for PosixDatastore (in particular) and a way to override that in repos without tons of repetition, and a way to easily create repos and Butler instances.

Here's a strawman for what the high-level interface could be:

Filesystem Structure

daf_butler

    config/

        butler.yaml # includes StorageClasses and schema

        gen2convert.yaml

        datastores/

            PosixDatastore.yaml # stuff specific to PosixDatastore (including formatters and templates). Does not include root.

        registries/

            SqlRegistry.yaml # stuff specific to SqlRegistry. Does not include DB connection string.

some-repo-root/

    butler.yaml # includes all entires from butler.yaml, datastores/.yaml, registries/.yaml, as well as repo-specific strings.

    registry3.sqlite3   # to avoid clashes with gen2 registry.sqlite3

APIs

Butler.makeRepo(root, registry=None, datastore=None):

Creates butler.yaml in root, copying in defaults from daf_butler for the given registry and datastore (which default to Sqlite and Posix), and adding root-specific entries to that config.

Butler(root):

Create Butler instance assuming makeRepo has already been called on the given root.

Config merging now works and I've had a go at handling the sub configs. One thing we should consider is changing RegistryConfig to refer to only the registry section and SchemaConfig only the schema section since they are now distinct. This would mean a small change to how Registry is configured, at least having two configs internally even if ButlerConfig is only way to create.

While I'm just starting to look at the code now, your proposal to make RegistryConfig and SchemaConfig refer only to their own sections sounds like a good one to me.

I added some more minor comments to the PR. Consider my review to be done. Feel free to merge with or without the suggested changes as you see fit.

My review is complete, too.  (Minor) comments on the PR.

Merged.

• Defaults read from DAF_BUTLER_DIR/config and DAF_BUTLER_CONFIG_PATH
• Each config subclass now reads its own defaults
• Butler config now asks each subclass config to populate defaults
• Configs can now be created without arguments to get the defaults.
• Extensive tests added for this behavior.
• Also rewrote doImport to use python3.6 features allowing useful error messages.