Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-30095

No transaction contribution bookkeeping made when ingesting files via the binary protocol

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      The bug

      The big is caused by the incomplete implementation of the feature explained in DM-26101.
      The worker-side ingest service that processes requests over the proprietary binary protocol (BOOST ASIO, Google Protobuf) is not recording contributions in the persistent state of the Replication/System when processing the request. This excludes contributions from being properly handled by the system at the transaction commit time and the catalog publishing time. Other algorithms (see DM-28603) relying on the persistent records for the contributions made during the transactions are also affected by the problem.

      The second improvement is to allow passing back to a client the retry_allowed flag that's meant to indicate if a failed contribution could be retried.

      NOTE: the file contributions will be recorded with file URLs adhering to this scheme https://en.wikipedia.org/wiki/File_URI_scheme

        Attachments

          Issue Links

          is triggered by

          Improvement - An improvement or enhancement to an existing feature or task. DM-26101 Add bookkeeping for table contributions requested during catalog ingests

          • Done

          Improvement - An improvement or enhancement to an existing feature or task. DM-28603 More efficient implementation of the Replication/Ingest algorithms in Qserv

          • Done

            Activity

            Ascending order - Click to sort in descending order
            Hide
            Permalink
            gapon Igor Gaponenko added a comment -

            PR: https://github.com/lsst/qserv/pull/621

            Show
            gapon Igor Gaponenko added a comment - PR: https://github.com/lsst/qserv/pull/621
            Hide
            Permalink
            npease Nate Pease added a comment -

            I don't think I'm capable of understanding the big picture here without a lot more discussion, but the changes look fine. There's one small nit about the language in a comment but otherwise looks 

            Show
            npease Nate Pease added a comment - I don't think I'm capable of understanding the big picture here without a lot more discussion, but the changes look fine. There's one small nit about the language in a comment but otherwise looks 

              People

              Assignee:
              gapon Igor Gaponenko
              Reporter:
              gapon Igor Gaponenko
              Reviewers:
              Nate Pease
              Watchers:
              Fabrice Jammes, Fritz Mueller, Hsin-Fang Chiang, Igor Gaponenko, Nate Pease
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  CI Builds

                  No builds found.