Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-10034

How to handle logging for a repeatedly-run subtask?

    XMLWordPrintable

    Details

    • Type: Story
    • Status: To Do
    • Resolution: Unresolved
    • Fix Version/s: None
    • Component/s: log
    • Labels:
      None
    • Team:
      Data Access and Database

      Description

      How should one handle limiting the logging of a subtask when it is run many times?

      This arose in DM-7611: a task can be run either once (in which case it generates a limited amount of useful INFO logging), or be run as a subtask, in which case it is run many (>100) times. In the latter case, the INFO logging is not as useful, and the calling task squashes the INFO logging of the subtask before calling its `run` method, but that could be dangerous if the user actually wants to get all of the INFO messages.

      Here is the discussion in the PR:
      https://github.com/lsst/ip_diffim/pull/53#discussion_r108770375

      What is the correct way to handle this?

        Attachments

          Issue Links

            Activity

            Hide
            tjenness Tim Jenness added a comment -

            Does anyone have any comment as to what the solution for this ticket might be? Maybe Yusra AlSayyad, Nate Lust, or Andy Salnikov have opinions as to how to deal with this or whether we should ignore it?

            Show
            tjenness Tim Jenness added a comment - Does anyone have any comment as to what the solution for this ticket might be? Maybe Yusra AlSayyad , Nate Lust , or Andy Salnikov have opinions as to how to deal with this or whether we should ignore it?
            Hide
            salnikov Andy Salnikov added a comment -

            I think setting logging level in the code is a bad practice indeed. Someone has to decide on logging strategy and I'm not sure if we can provide any decision for that. I believe we already have all tools that are needed to implement a reasonable strategy if someone else decides what it is.

            One random idea for a strategy - the task could have a flag which defines the logging level for the messages, you could select INFO (or default) if you run it once or DEBUG if you run it many times. If a user want to see messages from latter case then it's easy to change logging level from command line for that particular task (or whole thing).

            Show
            salnikov Andy Salnikov added a comment - I think setting logging level in the code is a bad practice indeed. Someone has to decide on logging strategy and I'm not sure if we can provide any decision for that. I believe we already have all tools that are needed to implement a reasonable strategy if someone else decides what it is. One random idea for a strategy - the task could have a flag which defines the logging level for the messages , you could select INFO (or default) if you run it once or DEBUG if you run it many times. If a user want to see messages from latter case then it's easy to change logging level from command line for that particular task (or whole thing).
            Hide
            ktl Kian-Tat Lim added a comment -

            I very much agree with Andy that setting the logging threshold in the code is a bad practice.

            I can empathize with the strong psychological pull to having some default logging from a Task, but rationally it seems to me that if a message should be DEBUG in the run-many-times context, it should still be DEBUG in the run-once context.

            Show
            ktl Kian-Tat Lim added a comment - I very much agree with Andy that setting the logging threshold in the code is a bad practice. I can empathize with the strong psychological pull to having some default logging from a Task, but rationally it seems to me that if a message should be DEBUG in the run-many-times context, it should still be DEBUG in the run-once context.
            Hide
            nlust Nate Lust added a comment -

            What about through config? Seems like this is something that is natural for pex_config.

            • Config base class for tasks defines a field that controls the logging for that task
            • There is a setting that controls the behavior for all subtasks that fires before the task config log level so you could override just one task
            • This allows Pipelines to control longing per label, in the case there is more than one instance of a task running
            • Allows individual control for a task if it is both a top level task and a sub task
            • log levels would then be stored with configuration (in config files, pipelines, persisted configs etc) and would be repeatable each time a task is run.
            • Lets logging for a task (and subtasks) be set at task creation time
            Show
            nlust Nate Lust added a comment - What about through config? Seems like this is something that is natural for pex_config. Config base class for tasks defines a field that controls the logging for that task There is a setting that controls the behavior for all subtasks that fires before the task config log level so you could override just one task This allows Pipelines to control longing per label, in the case there is more than one instance of a task running Allows individual control for a task if it is both a top level task and a sub task log levels would then be stored with configuration (in config files, pipelines, persisted configs etc) and would be repeatable each time a task is run. Lets logging for a task (and subtasks) be set at task creation time

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              reiss David Reiss
              Watchers:
              Andy Salnikov, David Reiss, Fritz Mueller, Hsin-Fang Chiang, John Parejko, Kian-Tat Lim, Nate Lust, Simon Krughoff, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

                Dates

                Created:
                Updated: