Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-10215

Add logging to Executor

    Details

      Description

      Currently Executor does not support any logging which makes tracking execution/debugging rather hard. The goal is to add a logger which would allow for easy monitoring of an Executor run.

        Attachments

          Activity

          mkowalik Mikolaj Kowalik created issue -
          mkowalik Mikolaj Kowalik made changes -
          Field Original Value New Value
          Epic Link DM-9662 [ 30730 ]
          mkowalik Mikolaj Kowalik made changes -
          Status To Do [ 10001 ] In Progress [ 3 ]
          Hide
          mkowalik Mikolaj Kowalik added a comment - - edited

          Could you please look at the changes I made to implement logging in Executor?

          For your convenience, here are setup steps if you feel like giving it a whirl (just remember to replace $DIR with actual path where you clone the repository to):

          $ setup pipe_tasks
          $ export PYTHONPATH="$PYTHONPATH:$DIR/Executor"
          $ export PATH="$PATH:$DIR/Executor/bin"
          $ execute -d job.json
          

          Option -d (--dry-run) will make Executor just to log the events without actually running the task so you shouldn't need anything else (you can find job.json in project's example directory).

          Show
          mkowalik Mikolaj Kowalik added a comment - - edited Could you please look at the changes I made to implement logging in Executor ? For your convenience, here are setup steps if you feel like giving it a whirl (just remember to replace $DIR with actual path where you clone the repository to): $ setup pipe_tasks $ export PYTHONPATH="$PYTHONPATH:$DIR/Executor" $ export PATH="$PATH:$DIR/Executor/bin" $ execute -d job.json Option -d ( --dry-run ) will make Executor just to log the events without actually running the task so you shouldn't need anything else (you can find job.json in project's example directory).
          mkowalik Mikolaj Kowalik made changes -
          Reviewers Hsin-Fang Chiang [ hchiang2 ]
          Status In Progress [ 3 ] In Review [ 10004 ]
          Hide
          hchiang2 Hsin-Fang Chiang added a comment -

          I tried the dryrun and could see the logging messages from Executor so it seems that's all intended in this ticket's descriptions, so marked this as Reviewed.

          I'm not able to actually run a pipeline. Naively I replaced the input root in job.json with a butler repo root, but it tried to remove the repo in this line. That's scary! Fortunately the input repo was protected so it gave me a permission error.

          At a later time, we might need integrations with the pipelines in terms of logging.

          Show
          hchiang2 Hsin-Fang Chiang added a comment - I tried the dryrun and could see the logging messages from Executor so it seems that's all intended in this ticket's descriptions, so marked this as Reviewed. I'm not able to actually run a pipeline. Naively I replaced the input root in job.json with a butler repo root, but it tried to remove the repo in this line . That's scary! Fortunately the input repo was protected so it gave me a permission error. At a later time, we might need integrations with the pipelines in terms of logging.
          hchiang2 Hsin-Fang Chiang made changes -
          Status In Review [ 10004 ] Reviewed [ 10101 ]
          Hide
          mkowalik Mikolaj Kowalik added a comment -

          Thanks for the prompt review and great point with removing the repo!

          I mean what it did was expected. If data section exists in the job specification, Executor assumes that it needs to create a brand new repo from scratch at the specified location and obliterates everything there prior to doing that. But now I clearly see I should definitely changed its behavior in favor of terminating the execution with an error in such cases.

          Show
          mkowalik Mikolaj Kowalik added a comment - Thanks for the prompt review and great point with removing the repo! I mean what it did was expected. If data section exists in the job specification, Executor assumes that it needs to create a brand new repo from scratch at the specified location and obliterates everything there prior to doing that. But now I clearly see I should definitely changed its behavior in favor of terminating the execution with an error in such cases.
          mkowalik Mikolaj Kowalik made changes -
          Resolution Done [ 10000 ]
          Status Reviewed [ 10101 ] Done [ 10002 ]
          frossie Frossie Economou made changes -
          Team Process Middleware [ 10206 ] Data Facility [ 12219 ]

            People

            • Assignee:
              mkowalik Mikolaj Kowalik
              Reporter:
              mkowalik Mikolaj Kowalik
              Reviewers:
              Hsin-Fang Chiang
              Watchers:
              Hsin-Fang Chiang, Mikolaj Kowalik
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Summary Panel