Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-12557

False positives when testing for pipeline completion

    Details

    • Templates:
    • Story Points:
      4
    • Team:
      Alert Production

      Description

      ap_pipe tests whether pipeline steps have been completed, and runs them only if they have not. However, errors in a pipeline step may cause it to appear completed when it is not. Two cases have been detected so far:

      • an AttributeError (from mismatched pipeline versions) in an obs package is silently ignored during ingestion, causing no data to be loaded. Re-running the pipeline skips data ingestion.
      • an exception during metadata handling causes a pipeline abort. Re-running the pipeline skips the step, so no metadata from it are available.

      It would be appreciated if the checks could be made a little more conservative, so that a step is only skipped if it is definitely finished.

        Attachments

          Container Issues

            Issue Links

              Activity

                People

                • Assignee:
                  mrawls Meredith Rawls
                  Reporter:
                  krzys Krzysztof Findeisen
                  Watchers:
                  Krzysztof Findeisen
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  1 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved:

                    Summary Panel