Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-4342

Disentangle log messages from different processes


    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: pex_logging, pipe_base
    • Labels:


      It's difficult to disentangle interleaved log messages when running with multiprocessing. Two possible ways to do this would be:
      1. Have each process write to a different log file. Maybe we could allow the user to specify log filenames including %(pid)s and %(hostname)s. More useful would be to allow the full range of keys from a dataId, but that might require some changes in how processing runs.
      2. Prepend each line of the log with context information (dataId, pid, hostname), allowing the user to use grep to do the disentangling. To avoid overwhelming the log with the context information, a hash of the context information could be used instead, with the lookups published in the log before the processing starts.

      Brief summary on the changes:
      • Chain logs of sub-tasks to their parent task logs
      • Add PrependedFormatter as the new default formatter for log files (specified through --logdest command line argument for cmdLineTask). The standard output to the terminal remains the same as before.
      • Any string can be used to label a pex_logging Log, and with PrependedFormatter this label prepends each log message. For cmdLineTask, the dataId is prepended.
      • The HSC's commit on prepending the timestamps is also ported to PrependedFormatter.
      • An example log message:

        2016-03-08T22:29:56.889933: [{'filter': 'r', 'tract': 0, 'patch': '2,3'}]: mergeCoaddDetections: Culled 1731 of 7751 peaks


          Issue Links



              • Assignee:
                hchiang2 Hsin-Fang Chiang
                price Paul Price
                Paul Price
                Andy Salnikov, David Nidever [X] (Inactive), Hsin-Fang Chiang, Jacek Becla, John Swinbank, Kian-Tat Lim, Margaret Gelman, Paul Price, Robert Lupton, Russell Owen
              • Votes:
                0 Vote for this issue
                10 Start watching this issue


                • Created:

                  Summary Panel