In order to support ingest of files from object store with any reasonable performance we need to be able to pre-calculate the ingest metadata and then use that metadata for ingest. Files will likely be ingested repeatedly and each time we do not want to have to download either the full file or the first N bytes to parse FITS headers.

Mofdify butler ingest-raws so that it can read JSON metadata snippet. This can either be in the form of a .json file of the same name as the data file, or optionally an update of ButlerURI to get it to read metadata from the object. Kian-Tat Lim has suggested that we can use the object metadata (see https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingMetadata.html but I'm not yet sure if the astronomy metadata fits in 2kB although we could probably encode it).

Pre-calculating this metadata will be a different ticket.

Since this is only really an issue for object stores, I am marking this as an IDF ticket.