Details
-
Type:
Story
-
Status: Won't Fix
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: daf_butler
-
Labels:
-
Story Points:2
-
Team:Data Release Production
Description
Running a query like
list(butler.registry.queryDatasets("calexp", collections=["shared/ci_hsc_output"],
|
skymap="discrete/ci_hsc", tract=0, patch=70))
|
produces many duplicate records for unclear reasons (they're not from having multiple collections, so deduplicate doesn't help). Try to fix this, or at least document when duplicate results must be expected.
Attachments
Issue Links
Activity
Is it worth doing this quickly as a separate ticket and reserving this ticket for understanding why the query itself is producing duplicates?
That sounds reasonable, though I may still not get to it super quickly. Others are welcome (Arun Kannawadi expressed in another daf_butler issue that turns out to be trickier than this one; maybe he'd be interested?)
Yes, I can work on it to remove the duplicates for now. Should I create another ticket? And is this a Gen3 butler?
This is gen3. Yes, I think a new ticket.
DM-22286 was not closed quickly. Is this ticket still an issue?
Since we know that all the DatasetRefs being returned must be coming from the same butler repository, this means that the ref.id must be unique. A short term fix is therefore to deduplicate the results by checking for duplicated ref.id – that's a couple of lines of code using a dict.
Is it worth doing this quickly as a separate ticket and reserving this ticket for understanding why the query itself is producing duplicates?