Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-9375

Revisit the Qserv neighbor-query radius (overlap distance)


    • Templates:
    • Team:
      DM Science


      In LDM-135, section 3.3.6, it says:

      To avoid [the need for data exchange among shard servers], each partition can be stored with a pre-computed amount of overlapping data. This overlapping data does not strictly belong to the partition but is within a preset spatial distance from the partition's borders. Using this data, spatial joins can be computed correctly within the preset distance without needing data from other partitions that may be on other nodes.

      Overlap is needed only for the Object Catalog, as all spatial correlations will be run on that catalog only. Guided by the experience from other projects including SDSS, we expect to preset the overlap to ~1 arcmin, which results in duplicating approximately 30% of the Object Catalog.

      This 1 arcmin overlap currently sets the scale for Qserv's ability to run neighboring-object queries.

      This specification goes back into the mists of LSST's early days and the reference queries that were used in the database design. With operations coming closer, and the first exposure of science data in Qserv to users as part of PDAC testing, it seems timely to take another look at this specification and check with science collaborations representing key LSST science use cases to ensure that we understand how it interacts with the work they expect to do.

      The goal of this epic is to identify several sample cases with SC partners in the next few months, assess the consequences of the 1 arcmin specification with them, and prepare a report on this issue that could be shared with the SAC and/or at a session at the August 2017 AHM.


          Issue Links



              • Assignee:
                ctslater Colin Slater
                gpdf Gregory Dubois-Felsmann
                Fritz Mueller, Gregory Dubois-Felsmann, Kian-Tat Lim, Leanne Guy
              • Votes:
                0 Vote for this issue
                4 Start watching this issue


                • Due:

                  Summary Panel