Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-28355

Allow ingest-raws to fail immediately on first error

    XMLWordPrintable

    Details

    • Story Points:
      1
    • Team:
      Architecture
    • Urgent?:
      No

      Description

      The Data Backbone Endpoint Buffer Manager's gen3 ingest code currently calls RawIngestTask.run() to ingest a single raw file at a time.  Upon failure to ingest, it tries to store an error message in a database for ease of monitoring and reporting.   Currently due to the batch nature of the gen3 middleware ingest code, the exception raised has a generic message (Some failures encountered during ingestion) and details are output in the debug log.  It would be helpful if there could be some kind of fail early/first switch that if set would raise on first error and have a more specific error message in the exception.  The debug logging will always be able to have more details.  But some way to group errors (or to show same error across ingestion versions) in the DBBBM database records would be useful.

        Attachments

          Activity

          Hide
          tjenness Tim Jenness added a comment -

          Change was fairly simple. I'm not sure how to test it though since obs_base doesn't have any test instrument data that could be bad and which would trigger the tests. I have tried each of the conditions from the command line. You can add -c failFast=True to try it out with that.

          Show
          tjenness Tim Jenness added a comment - Change was fairly simple. I'm not sure how to test it though since obs_base doesn't have any test instrument data that could be bad and which would trigger the tests. I have tried each of the conditions from the command line. You can add -c failFast=True to try it out with that.
          Hide
          mgower Michelle Gower added a comment -

          Since Mikolaj Kowalik's been working on the code that would use this, changing the reviewer to him.

          Show
          mgower Michelle Gower added a comment - Since Mikolaj Kowalik 's been working on the code that would use this, changing the reviewer to him.
          Hide
          mkowalik Mikolaj Kowalik added a comment - - edited

          Looks good to me.

          And some preliminary tests I made show that we got exactly what we asked for. Thanks for making it happen.

          Show
          mkowalik Mikolaj Kowalik added a comment - - edited Looks good to me. And some preliminary tests I made show that we got exactly what we asked for. Thanks for making it happen.

            People

            Assignee:
            tjenness Tim Jenness
            Reporter:
            mgower Michelle Gower
            Reviewers:
            Mikolaj Kowalik
            Watchers:
            Michelle Gower, Mikolaj Kowalik, Tim Jenness
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                Jenkins Builds

                No builds found.