Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-3472

Implement Basic spatial lookups for the Butler

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: butler
    • Labels:
      None
    • Story Points:
      10
    • Epic Link:
    • Sprint:
      DB_W16_11, DB_W16_12, DB_W16_01, DB_W16_02, DB_W16_03, DB_X16_04, DB_X16_03
    • Team:
      Data Access and Database

      Description

      Based on this pipeline software will require spatial lookups via butler.

      Current plan is:
      Provide an "ingest" script that looks at a butler exposure dataset, pulls spatial information and data ID out of it, then shoves all of that into an sqlite3 table that can be used for accelerated spatial lookups.
      Then provide a task that takes skymap parameters, and maps those to potentially overlapping exposures: the sqlite3 table is outside butler; and the will use a spatial location to get a dataId, and then pass that dataId into butler to get that file.

        Attachments

          Issue Links

            Activity

            Hide
            smonkewitz Serge Monkewitz added a comment -

            Concerning the review request to use Box2I.getCorners - it looks like that would give me the pixel indexes of the exposure bounding box. I then need to map those to pixel positions (using afw_image.indexToPosition) before I can call Wcs.pixelToSky, right? If I'm understanding things, indexToPosition would get me a bounding box for the positions of pixel centers. Should I then treat pixels as having a spatial extent (i.e. pad that bounding box by 0.5 pixels as I am doing now), or should I treat them like point samples?

            I'm guessing a 0.5 pixel adjustment isn't going to be very noticeable, but I'd like to understand what the correct thing to do is.

            A related idea would be a configureable pixel bounding box padding value. You might for example set it to a negative value -N, so that if you subsequently find an intersection between the exposure bounds and a search region, you know that search region got at least N pixel into the exposure. It would also be a way to compensate for projection and/or distortion related errors. Worth adding?

            Show
            smonkewitz Serge Monkewitz added a comment - Concerning the review request to use Box2I.getCorners - it looks like that would give me the pixel indexes of the exposure bounding box. I then need to map those to pixel positions (using afw_image.indexToPosition ) before I can call Wcs.pixelToSky , right? If I'm understanding things, indexToPosition would get me a bounding box for the positions of pixel centers. Should I then treat pixels as having a spatial extent (i.e. pad that bounding box by 0.5 pixels as I am doing now), or should I treat them like point samples? I'm guessing a 0.5 pixel adjustment isn't going to be very noticeable, but I'd like to understand what the correct thing to do is. A related idea would be a configureable pixel bounding box padding value. You might for example set it to a negative value -N, so that if you subsequently find an intersection between the exposure bounds and a search region, you know that search region got at least N pixel into the exposure. It would also be a way to compensate for projection and/or distortion related errors. Worth adding?
            Hide
            price Paul Price added a comment -

            We treat pixels as point samples.

            I like the idea of the padding value, but I'm worried about adding scope especially in light of your schedule.

            Show
            price Paul Price added a comment - We treat pixels as point samples. I like the idea of the padding value, but I'm worried about adding scope especially in light of your schedule.
            Hide
            smonkewitz Serge Monkewitz added a comment -

            Can you take another look? I think I've addressed everything. I ended up changing batch_size to defer_writes, which defers writes to the runner. Currently they are done as one big transaction, which I don't think should be a problem. But if I'm wrong, it'd be easy enough to introduce a knob.

            Show
            smonkewitz Serge Monkewitz added a comment - Can you take another look? I think I've addressed everything. I ended up changing batch_size to defer_writes , which defers writes to the runner. Currently they are done as one big transaction, which I don't think should be a problem. But if I'm wrong, it'd be easy enough to introduce a knob.
            Hide
            price Paul Price added a comment -

            Awesome!

            The only suggestion I have (take it or leave it at your pleasure) is that maybe defer_writes should be a command-line argument rather than a config parameter, since it's an operational concern.

            Show
            price Paul Price added a comment - Awesome! The only suggestion I have (take it or leave it at your pleasure) is that maybe defer_writes should be a command-line argument rather than a config parameter, since it's an operational concern.
            Hide
            smonkewitz Serge Monkewitz added a comment -

            I'm going to leave it for now due to time constraints, and since it can be set via a command line config override. If you feel strongly about it, please file an issue so it isn't forgotten. Thanks for the speedy review!

            Show
            smonkewitz Serge Monkewitz added a comment - I'm going to leave it for now due to time constraints, and since it can be set via a command line config override. If you feel strongly about it, please file an issue so it isn't forgotten. Thanks for the speedy review!

              People

              Assignee:
              smonkewitz Serge Monkewitz
              Reporter:
              fritzm Fritz Mueller
              Reviewers:
              Paul Price
              Watchers:
              Fritz Mueller, Jacek Becla, John Parejko, Nate Pease [X] (Inactive), Paul Price, Serge Monkewitz
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.