Details
-
Type:
Story
-
Status: Done
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: butler
-
Labels:None
-
Story Points:10
-
Epic Link:
-
Sprint:DB_W16_11, DB_W16_12, DB_W16_01, DB_W16_02, DB_W16_03, DB_X16_04, DB_X16_03
-
Team:Data Access and Database
Description
Based on this pipeline software will require spatial lookups via butler.
Current plan is:
Provide an "ingest" script that looks at a butler exposure dataset, pulls spatial information and data ID out of it, then shoves all of that into an sqlite3 table that can be used for accelerated spatial lookups.
Then provide a task that takes skymap parameters, and maps those to potentially overlapping exposures: the sqlite3 table is outside butler; and the will use a spatial location to get a dataId, and then pass that dataId into butler to get that file.
Attachments
Issue Links
Activity
We treat pixels as point samples.
I like the idea of the padding value, but I'm worried about adding scope especially in light of your schedule.
Paul Price
added a comment - We treat pixels as point samples.
I like the idea of the padding value, but I'm worried about adding scope especially in light of your schedule. Can you take another look? I think I've addressed everything. I ended up changing batch_size to defer_writes, which defers writes to the runner. Currently they are done as one big transaction, which I don't think should be a problem. But if I'm wrong, it'd be easy enough to introduce a knob.
Serge Monkewitz
added a comment - Can you take another look? I think I've addressed everything. I ended up changing batch_size to defer_writes , which defers writes to the runner. Currently they are done as one big transaction, which I don't think should be a problem. But if I'm wrong, it'd be easy enough to introduce a knob. Awesome!
The only suggestion I have (take it or leave it at your pleasure) is that maybe defer_writes should be a command-line argument rather than a config parameter, since it's an operational concern.
Paul Price
added a comment - Awesome!
The only suggestion I have (take it or leave it at your pleasure) is that maybe defer_writes should be a command-line argument rather than a config parameter, since it's an operational concern. I'm going to leave it for now due to time constraints, and since it can be set via a command line config override. If you feel strongly about it, please file an issue so it isn't forgotten. Thanks for the speedy review!
Serge Monkewitz
added a comment - I'm going to leave it for now due to time constraints, and since it can be set via a command line config override. If you feel strongly about it, please file an issue so it isn't forgotten. Thanks for the speedy review! 
Concerning the review request to use Box2I.getCorners - it looks like that would give me the pixel indexes of the exposure bounding box. I then need to map those to pixel positions (using afw_image.indexToPosition) before I can call Wcs.pixelToSky, right? If I'm understanding things, indexToPosition would get me a bounding box for the positions of pixel centers. Should I then treat pixels as having a spatial extent (i.e. pad that bounding box by 0.5 pixels as I am doing now), or should I treat them like point samples?
I'm guessing a 0.5 pixel adjustment isn't going to be very noticeable, but I'd like to understand what the correct thing to do is.
A related idea would be a configureable pixel bounding box padding value. You might for example set it to a negative value -N, so that if you subsequently find an intersection between the exposure bounds and a search region, you know that search region got at least N pixel into the exposure. It would also be a way to compensate for projection and/or distortion related errors. Worth adding?