Details
-
Type:
Story
-
Status: Done
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: daf_butler
-
Labels:
-
Story Points:1
-
Epic Link:
-
Team:Data Release Production
-
Urgent?:No
Description
This query returns incorrectly returns no results:
$ butler query-datasets /project/hsc/gen3repo/rc2w02_ssw03 bfKernel --collections='*' |
even though this query succeeds:
$ butler query-datasets /project/hsc/gen3repo/rc2w02_ssw03 'bfKernel' --collections=HSC/calib/unbounded
|
The latter query certainly felt much slower than it should. We should at least profile it.
Something seems to be going wrong with the logic that attempts to query all RUN collections (and only RUN collections) when the collections are unconstrained.
Attachments
Activity
| Field | Original Value | New Value |
|---|---|---|
| Description |
This query returns incorrectly returns no results:
{code:java} $ butler query-datasets /project/hsc/gen3repo/rc2w02_ssw03 bfKernel --collections='*' {code} even though this query succeeds: {code} $ butler query-datasets /project/hsc/gen3repo/rc2w02_ssw03 'bfKernel' --collections=HSC/calib/unbounded {code} Something seems to be going wrong with the logic that attempts to query all RUN collections (and only RUN collections) when the collections are unconstrained. |
This query returns incorrectly returns no results:
{code:java} $ butler query-datasets /project/hsc/gen3repo/rc2w02_ssw03 bfKernel --collections='*' {code} even though this query succeeds: {code} $ butler query-datasets /project/hsc/gen3repo/rc2w02_ssw03 'bfKernel' --collections=HSC/calib/unbounded {code} The latter query certainly _felt_ much slower than it should. We should at least profile it. Something seems to be going wrong with the logic that attempts to query all RUN collections (and only RUN collections) when the collections are unconstrained. |
| Status | To Do [ 10001 ] | In Progress [ 3 ] |
| Reviewers | Nate Pease [ npease ] | |
| Status | In Progress [ 3 ] | In Review [ 10004 ] |
no problem re. reviews.
the logic seems fine. There's a small change you can make to simply code, noted in the PR.
| Status | In Review [ 10004 ] | Reviewed [ 10101 ] |
| Resolution | Done [ 10000 ] | |
| Status | Reviewed [ 10101 ] | Done [ 10002 ] |
| Story Points | 1 |
| Epic Link | DM-27956 [ 442730 ] |
Nate Pease [X], sorry about hitting you up for back-to-back reviews, but this one is also small and even more in "your" part of daf_butler, so I'd like to make sure this isn't going in what you'd consider the wrong direction.
See the (only) commit message re what the problem was and why I fixed it this way (also the PR description).
As for the performance aspect of the ticket description, I did some profiling and it's totally dominated by butler startup costs (Python imports and aggressive fetching from the DB in particular). So while that's not great, and something for us to look out for, it's not easily fixed and hence not something I'm going to bother with on this ticket.