Details
-
Type:
Story
-
Status: Done
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: daf_butler
-
Labels:None
-
Story Points:2
-
Team:Data Access and Database
-
Urgent?:No
Description
In DM-25180 we added ingest time to the datasets table. Now we need to be able to include the ingest time in a queryDatasets call to allow a query to retrieve all the files ingested between two dates. This will allow OODS to be able to clear out old files without having to know about datastore implementations.
Attachments
Issue Links
- is triggered by
-
- Done
Activity
I imagine OODS doesn't care about datasetType and just wants where ingest_date < 'some_date'. I could definitely though envisage being able to purge by dataset type so that you can delete all the PVIs ingested before a date. Steve Pietrowicz can confirm about the need for datasetType for OODS though.
I think this is ready for review. Implementation is not too terrible but it has some dialect-specific code. I tested it with both SQLite and PostgreSQL, seems to be working OK.
A few minor comments on the PR.
Thanks for review and suggestions, unit test moved, merged.
For some other query expressions involving datasets (e.g. the timespan associated with a dataset in a CALIBRATION collection), I was planning to use the dataset type name as a table-like identifier, and map that in ClauseVisitor to the subquery that searches for that dataset type in the given sequence of collections. So I think that's a good choice if we did want to allow ingest time filtering on just some dataset types. I don't know how much it informs what we should do if we always want to do this filtering on all dataset types, which is what I think is more likely for OODS.