ap_pipe tests whether pipeline steps have been completed, and runs them only if they have not. However, errors in a pipeline step may cause it to appear completed when it is not. Two cases have been detected so far:
- an AttributeError (from mismatched pipeline versions) in an obs package is silently ignored during ingestion, causing no data to be loaded. Re-running the pipeline skips data ingestion.
- an exception during metadata handling causes a pipeline abort. Re-running the pipeline skips the step, so no metadata from it are available.
It would be appreciated if the checks could be made a little more conservative, so that a step is only skipped if it is definitely finished.