XMLWordPrintable

#### Details

• Type: Story
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s: None
• Labels:
• Story Points:
2
• Team:
Data Facility

#### Description

Currently Executor does not support any logging which makes tracking execution/debugging rather hard. The goal is to add a logger which would allow for easy monitoring of an Executor run.

#### Activity

Hide
Mikolaj Kowalik added a comment - - edited

Could you please look at the changes I made to implement logging in Executor?

For your convenience, here are setup steps if you feel like giving it a whirl (just remember to replace $DIR with actual path where you clone the repository to):  $ setup pipe_tasks $export PYTHONPATH="$PYTHONPATH:$DIR/Executor" $ export PATH="$PATH:$DIR/Executor/bin" $execute -d job.json  Option -d (--dry-run) will make Executor just to log the events without actually running the task so you shouldn't need anything else (you can find job.json in project's example directory). Show Mikolaj Kowalik added a comment - - edited Could you please look at the changes I made to implement logging in Executor ? For your convenience, here are setup steps if you feel like giving it a whirl (just remember to replace$DIR with actual path where you clone the repository to): $setup pipe_tasks$ export PYTHONPATH="$PYTHONPATH:$DIR/Executor" $export PATH="$PATH:$DIR/Executor/bin"$ execute -d job.json Option -d ( --dry-run ) will make Executor just to log the events without actually running the task so you shouldn't need anything else (you can find job.json in project's example directory).
Hide
Hsin-Fang Chiang added a comment -

I tried the dryrun and could see the logging messages from Executor so it seems that's all intended in this ticket's descriptions, so marked this as Reviewed.

I'm not able to actually run a pipeline. Naively I replaced the input root in job.json with a butler repo root, but it tried to remove the repo in this line. That's scary! Fortunately the input repo was protected so it gave me a permission error.

At a later time, we might need integrations with the pipelines in terms of logging.

Show
Hsin-Fang Chiang added a comment - I tried the dryrun and could see the logging messages from Executor so it seems that's all intended in this ticket's descriptions, so marked this as Reviewed. I'm not able to actually run a pipeline. Naively I replaced the input root in job.json with a butler repo root, but it tried to remove the repo in this line . That's scary! Fortunately the input repo was protected so it gave me a permission error. At a later time, we might need integrations with the pipelines in terms of logging.
Hide
Mikolaj Kowalik added a comment -

Thanks for the prompt review and great point with removing the repo!

I mean what it did was expected. If data section exists in the job specification, Executor assumes that it needs to create a brand new repo from scratch at the specified location and obliterates everything there prior to doing that. But now I clearly see I should definitely changed its behavior in favor of terminating the execution with an error in such cases.

Show
Mikolaj Kowalik added a comment - Thanks for the prompt review and great point with removing the repo! I mean what it did was expected. If data section exists in the job specification, Executor assumes that it needs to create a brand new repo from scratch at the specified location and obliterates everything there prior to doing that. But now I clearly see I should definitely changed its behavior in favor of terminating the execution with an error in such cases.

#### People

Assignee:
Mikolaj Kowalik
Reporter:
Mikolaj Kowalik
Reviewers:
Hsin-Fang Chiang
Watchers:
Hsin-Fang Chiang, Mikolaj Kowalik