Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-5365

Enable CC-IN2P3/Qserv team communication in order to prepare for Pan-STARRS large scale tests

    Details

    • Type: Story
    • Status: Invalid
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: Qserv
    • Labels:
      None
    • Templates:
    • Story Points:
      8
    • Team:
      Data Access and Database

      Description

      The goal of this ticket is to enable communication between CC-IN2P3 and Qserv team in order to prepare for Pan-STARRS data ingestion into Qserv. This data ingestion step is necessary for the large scale tests of Qserv foreseen for summer 2016.

      Specifically, we need to understand:

      • What is the size of the data set to be imported to CC-IN2P3?
      • Where the Pan-STARRS data set to be imported is currently located?
      • What mechanisms will the host of Pan-STARRS data make available to CC-IN2P3 for downloading the data set?
      • Does the envisaged ingestion mechanism into Qserv requires that the data transit through the Qserv master server or will each Qserv worker be able to ingest its own chunk of data?
      • After the ingestion process is finished, do we need to keep a copy of the ingested data out of Qserv?

      Given the size of the dataset likely involved in this process, this project will probably require that we (both Qserv and CC-IN2P3 experts) set up specific mechanisms and equipment for efficient transport, storage and ingestion of these data. Timely planning and several testing campaigns seem necessary for this project to make progress.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                jammes Fabrice Jammes
                Watchers:
                Andy Salnikov, Fabio Hernandez, Fabrice Jammes, Fritz Mueller, Jacek Becla, Serge Monkewitz, Yvan Calas
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel