Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-15877

Read and comment on DMTN-090

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: Design Documents
    • Labels:
      None
    • Story Points:
      1
    • Sprint:
      Arch 2018-10-01, Arch 2018-10-08, Arch 2018-10-15
    • Team:
      Architecture

      Attachments

        Issue Links

          Activity

          Hide
          tjenness Tim Jenness added a comment -

          Christine Banek this is a good summary of the situation. Some minor comments:

          • I think the query history has no expiry date in our requirements. For our performance monitoring we will need to record query times and associated parameters along with the query. Staff should be able to access the ensemble of queries otherwise it's impossible to use satisfy the auditing or improving the system based on usage modes.
          • I think users would expect that they can store query results as long as they want so long as they fit into their allocated storage space. Effectively this means an ability to move the results into their VOSpace area and rejecting the query if it doesn't fit in there. The standard allows us to write whatever format we want to the VOSpace area (writing VOTable if requested by the client) so we could write a binary results file and give people the code they need to read it in the science platform if that is a more common use case.
          • Do shared scans in Qserv allow us to keep more queries in flight than sync queries to Qserv?
          • It's probably a good idea to log SODA/SIA queries. I doubt we need that history to be accessible to users.
          • SODA could also presumably write the results to VOSpace.
          • Technically the butler is retrieving the data but the PipelineTasks are processing it.
          • Regarding deployment, we are required to be able to do queries and retrievals on DR1 using the version of the services compatible with DR1. So if you had a query that worked for DR1, ten years later that query should still gave the same results for DR1. We don't know whether that mans that the DR11 version of the software has to handle DR1 queries or if we have an instance of the DR1 system that's been hanging around for 10 years connecting to DR1 databases...
          • Qserv has a bunch of predefined standard queries. These should be ported to ADQL as part of the test suite.
          Show
          tjenness Tim Jenness added a comment - Christine Banek this is a good summary of the situation. Some minor comments: I think the query history has no expiry date in our requirements. For our performance monitoring we will need to record query times and associated parameters along with the query. Staff should be able to access the ensemble of queries otherwise it's impossible to use satisfy the auditing or improving the system based on usage modes. I think users would expect that they can store query results as long as they want so long as they fit into their allocated storage space. Effectively this means an ability to move the results into their VOSpace area and rejecting the query if it doesn't fit in there. The standard allows us to write whatever format we want to the VOSpace area (writing VOTable if requested by the client) so we could write a binary results file and give people the code they need to read it in the science platform if that is a more common use case. Do shared scans in Qserv allow us to keep more queries in flight than sync queries to Qserv? It's probably a good idea to log SODA/SIA queries. I doubt we need that history to be accessible to users. SODA could also presumably write the results to VOSpace. Technically the butler is retrieving the data but the PipelineTasks are processing it. Regarding deployment, we are required to be able to do queries and retrievals on DR1 using the version of the services compatible with DR1. So if you had a query that worked for DR1, ten years later that query should still gave the same results for DR1. We don't know whether that mans that the DR11 version of the software has to handle DR1 queries or if we have an instance of the DR1 system that's been hanging around for 10 years connecting to DR1 databases... Qserv has a bunch of predefined standard queries. These should be ported to ADQL as part of the test suite.

            People

            • Assignee:
              tjenness Tim Jenness
              Reporter:
              ktl Kian-Tat Lim
              Watchers:
              Gregory Dubois-Felsmann, Kian-Tat Lim, Tim Jenness
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Summary Panel