Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-12557

False positives when testing for pipeline completion

    Details

    • Templates:
    • Story Points:
      4
    • Team:
      Alert Production

      Description

      ap_pipe tests whether pipeline steps have been completed, and runs them only if they have not. However, errors in a pipeline step may cause it to appear completed when it is not. Two cases have been detected so far:

      • an AttributeError (from mismatched pipeline versions) in an obs package is silently ignored during ingestion, causing no data to be loaded. Re-running the pipeline skips data ingestion.
      • an exception during metadata handling causes a pipeline abort. Re-running the pipeline skips the step, so no metadata from it are available.

      It would be appreciated if the checks could be made a little more conservative, so that a step is only skipped if it is definitely finished.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                mrawls Meredith Rawls
                Reporter:
                krzys Krzysztof Findeisen
                Watchers:
                Krzysztof Findeisen
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel