Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-27091

Extend the Ingest system to allow pulling table contributions from a remote store

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: Qserv
    • Labels:
      None

      Description

      Goals

      Implement the HTTP-based ingest service at workers which would support uploading table contributions from a remote HTTP service using a URL provided by a client. Types of URLs supported by the new ingest service include:

      file:///<path>
      http://<resource>
      https://<resource>
      

      The file-based resources are expected to refer to files residing at filesystems (local or remote, such as GPFS, Luster, etc.) mounted on the worker nodes. Operations with the HTTP-based resources would be equivalent to:

      curl <url> -X <method> -H <header> -d <data>
      

      The newly added worker REST service would allow passing all four (of which only the method and url ones are required) parameters to the service.

      Modifications and improvements to the existing code

      • extend Configuration service of workers with DNS hostname and port number of the worker's HTTP service
      • refactor existing code of the Ingest server to support uploading table data from multiple sources
      • refactor Http modules (which are presently used by HttpProcessor) to allow using base functionality of the modules in other contexts
      • implement the Http-based worker ingest service
      • extent the REST metadata services of the Master Replication Controller to return DNS names and port numbers of the HTTP-based worker ingest services

      Other tasks:

      • update and extend documentation
      • bump the Ingest API version number to 2.
      • test the new ingest interface at a scale

      Direct links to the relevant documentation pages which hasve been updated, or added:

        Attachments

          Issue Links

            Activity

            Show
            gapon Igor Gaponenko added a comment - PR: https://github.com/lsst/qserv/pull/573
            Hide
            fritzm Fritz Mueller added a comment -

            Looks pretty good overall. A couple comments:

            • Configuration system has way too much boilerplate and repetition, and needs greater abstraction. This will need to be addressed in a future refactoring pass.
            • Not a fan at all of the lazy-instantiation changes made here, for reasons I've gone into before. I understand this is temporary pending an already-planned refactor to separate services from ServiceProvider.

            Okay to merge these changes as-is for now from my point-of-view, in order to unblock Fabrice.

            Show
            fritzm Fritz Mueller added a comment - Looks pretty good overall. A couple comments: Configuration system has way too much boilerplate and repetition, and needs greater abstraction. This will need to be addressed in a future refactoring pass. Not a fan at all of the lazy-instantiation changes made here, for reasons I've gone into before. I understand this is temporary pending an already-planned refactor to separate services from ServiceProvider. Okay to merge these changes as-is for now from my point-of-view, in order to unblock Fabrice.

              People

              Assignee:
              gapon Igor Gaponenko
              Reporter:
              gapon Igor Gaponenko
              Reviewers:
              Fritz Mueller
              Watchers:
              Fabrice Jammes, Fritz Mueller, Hsin-Fang Chiang, Igor Gaponenko, Nate Pease
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  CI Builds

                  No builds found.