Could you please look at the changes I made to implement logging in Executor?

For your convenience, here are setup steps if you feel like giving it a whirl (just remember to replace $DIR with actual path where you clone the repository to):

$ setup pipe_tasks

$ export PYTHONPATH="$PYTHONPATH:$DIR/Executor"

$ export PATH="$PATH:$DIR/Executor/bin"

$ execute -d job.json

Option -d (--dry-run) will make Executor just to log the events without actually running the task so you shouldn't need anything else (you can find job.json in project's example directory).

I tried the dryrun and could see the logging messages from Executor so it seems that's all intended in this ticket's descriptions, so marked this as Reviewed.

I'm not able to actually run a pipeline. Naively I replaced the input root in job.json with a butler repo root, but it tried to remove the repo in this line. That's scary! Fortunately the input repo was protected so it gave me a permission error.

At a later time, we might need integrations with the pipelines in terms of logging.

Thanks for the prompt review and great point with removing the repo!

I mean what it did was expected. If data section exists in the job specification, Executor assumes that it needs to create a brand new repo from scratch at the specified location and obliterates everything there prior to doing that. But now I clearly see I should definitely changed its behavior in favor of terminating the execution with an error in such cases.