Quentin Le Boulc'h reported the following problem when exporting calibrations from the DC2 repo:

$ butler export-calibs /repo/dc2 2.2i_calib_export  2.2i/calib/gen2

...

FileExistsError: Destination path 'file:///home/quentinleboulch/2.2i_calib_export/2.2i/calib/gen2/20220101T000000Z/bias/bias_LSSTCam-imSim_2_2i_calib_gen2_20220101T000000Z.fits' already exists. Transfer from file:///datasets/DC2/DR6/Run2.2i/patched/2021-02-10/CALIB/bias/2022-01-01/bias-R01-S01-det001_2022-01-01.fits cannot be completed.

There are a few problems but the fundamental one is that the DatasetRef being returned from queryDatasetAssociations are not fully expanded. This might not be a problem in itself but the way the export is setup triggers unexpected failures.

• As a debugging aid it would be helpful if export-calibs allowed the user to specify the dataset type they wanted in addition to the calibration collection.
• There is no way to set the transfer mode in this command so you always end up with auto.
• If you wanted to disable the file export but just get the XML with paths relative to the datastore there is no way to do it since an output directory is always created.
• "ingest" has logic to handle a "split" ingest transfer where some files are relative paths and some are absolute paths. Export does not know anything about this.
• In the absence of a "split" transfer option, when the exporter encounters absolute URIs in direct mode it decides its only course of action is to create local copies in the output directory. To do that it needs to build a file name from the dataset ref.
• The template wants to use the full detector name in the filename and therefore needs the detector record. Since the DatasetRef are not fully expanded it can't do that and silently leaves it out of the template.
• Without a detector in the filename the files are not unique and so everything breaks.

This suggests many fixes should be made to deal with this mess:

• Add a transfer option to the command.
• Add a datasets option to the command.
• Add support for "split" transfer mode to exportDatasets which would transfer files in the datastore but leave absolute URIs to files outside unchanged.
• Expand the DatasetRef retrieved by queryDatasetAssociations in case they will be used to copy the file from absolute URI to new location.
• Consider making FileTemplate complain if a dimension record is needed but none is available.