per Tim Jenness: I think the repo subset one can probably be folded into making transfer-datasets also transfer the dimension it needs. That is effectively moving some code out of execution butler creation into more reusable form. I think that would go a long way to allowing a subset of data to be directly transferred from one repo to another. A completely generic butler exporter command-line is much harder.

The changes were fairly simple but much of the logic was copied from export/import code. I did think about a Butler.transfer_dimension_data(refs) API (which would be called inside the transaction in Butler.transfer_from) but was not really sure it was worth doing.

For the butler transfer-datasets command I also wasn't sure whether transferring dimension records should be optional. In theory the batch system knows it's transferring back datasets that must exist in the destination so it would be a bit more efficient to disable the record copying. Most users are going to want it to be on by default. For now I've made it always do it but I can add a --transfer-dimensions flag if necessary (although it should be defaulted to true so --no-transfer-dimensions is what bps would have to use).

The change in pipe_base is to remove the code that explicitly exports/imports the dimension records – maybe that will speed up execution butler creation a little.

daf_butler changes look good, but I think a flag to not transfer dimension data for use by BPS is well worth the effort

pipe_base looks fine, too, but there's no PR. If you make it I'll approve it.

Krzysztof Findeisen hopefully this will help you with transferring your test datasets around without having to export and import the dimensions separately.