Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-27772

Improved error reporting in the ingest worker for failures when opening/reading table contributions

    XMLWordPrintable

    Details

    • Type: Improvement
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: Qserv
    • Labels:
      None

      Description

      Goals

      This development was triggered by DM-27091.
      The current version of the ingest system has a very simple error reporting mechanism for failures when opening table contribution files located at workers' filesystems. Locations of these files are passed to workers using this URL scheme:

      file:///<local-file-path>
      

      Any errors are reported to a client as a generic error message:

      failed to open file '/<local-file-path>'.
      

      Though, it's often desired to identify an actual cause of such failures. In many cases, problems would arise from non-existing files, the overly-tightened file access settings for the corresponding folders/files, or a filesystem not mounted into the worker's Docker container.

      In this ticket, the error handling mechanism will be improved to return the actual error reported by the underlying operating system (specifically by obtaining and translating errno). This will feed more information back to a client (that one initiated the file ingest) or a data administrator (responsible for setting up and managing Qserv) that may help to easier identify and fix the root cause of the problem. After implementing the proposed improvement, error messages reported to a client would look like:

      failed to open file '/<local-file-path>', error: 'No such file or directory', errno: 2
      failed to open file '/<local-file-path>', error: 'Permission denied', errno: 13
      ...
      

      Other improvements

      Improve error handling (and reporting) when pulling files to workers from an object store over HTTP/HTTPS protocol. The current version of the code doesn't treat HTTP error codes at 400 or greater as errors. This results in a confusing behavior of the worker ingest services after it gets (and tries to ingest into MySQL) an HTML document with the following content:

      <html>
      <head><title>404 Not Found</title></head>
      <body style="background-color:#E6E6FA">
      <h1>404 Not Found</h1>
      </body>
      </html>
      

        Attachments

          Issue Links

            Activity

            Show
            gapon Igor Gaponenko added a comment - PR: https://github.com/lsst/qserv/pull/585
            Hide
            salnikov Andy Salnikov added a comment -

            Looks good.

            Show
            salnikov Andy Salnikov added a comment - Looks good.

              People

              Assignee:
              gapon Igor Gaponenko
              Reporter:
              gapon Igor Gaponenko
              Reviewers:
              Andy Salnikov
              Watchers:
              Andy Salnikov, Fritz Mueller, Hsin-Fang Chiang, Igor Gaponenko, Nate Pease
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: