Fix Version/s: None
Team:Data Access and Database
Existing butler unit tests should run without an sqlite database registry.
- is triggering
DM-4693 undeclared dependencies on astropy.io and yaml
As I understand it, the Butler currently requires that alongside data there be an sqlite db that contains information about what data is there. To satisfy this story the Butler would look in the repository location, find the data sources by file name and use that found information as a registry (thus eliminating the need/requirement for an sqlite db).
I believe the registry is not strictly necessary, but only required for look-ups when there's insufficient information. For example, with HSC I often want to select CCDs by "field" and "filter", and the registry is used to provide the list of "visit", "ccd" and other information required to satisfy the filenames.
I believe the LSST ImSim data doesn't use a registry, and the user just has to provide all the required information to satisfy the filenames.
One of the things KT & discussed when he was giving me the crash course was to use python's glob function to add lookup, so maybe the point of this story is the wildcard lookup. Probably Kian-Tat Lim should weigh in.
There are two aspects to this. The first is lookup of keys not provided in the dataId, a function which pertains to both get() calls with less-than-complete information and queryMetadata() calls that almost always have less-than-complete information. All obs_* packages, including obs_lsstSim, currently use the registry for that second purpose. The second aspect, which is more complex and which I hadn't yet sprung on Nate, is reading information (in particular the observation time and length) out of an input dataset's file representation in order to provide rendezvous with calibration data in another repository (that does have a registry). Today, that read is handled by genInputRegistry.py so that the butler doesn't need to look into the dataset itself. If there's no registry, such a read will be necessary.
Some comments in the PR in daf_butlerUtils. The obs_decam change looks OK to me. As I understand it, no change will be made to obs_test, and the spurious "
DM-3591" branch in obs_decam will also be removed.
Could you explain what you mean, please? The butler already supports data repos without a registry.