It's difficult to disentangle interleaved log messages when running with multiprocessing. Two possible ways to do this would be:
1. Have each process write to a different log file. Maybe we could allow the user to specify log filenames including %(pid)s and %(hostname)s. More useful would be to allow the full range of keys from a dataId, but that might require some changes in how processing runs.
2. Prepend each line of the log with context information (dataId, pid, hostname), allowing the user to use grep to do the disentangling. To avoid overwhelming the log with the context information, a hash of the context information could be used instead, with the lookups published in the log before the processing starts.
- Chain logs of sub-tasks to their parent task logs
- Add PrependedFormatter as the new default formatter for log files (specified through --logdest command line argument for cmdLineTask). The standard output to the terminal remains the same as before.
- Any string can be used to label a pex_logging Log, and with PrependedFormatter this label prepends each log message. For cmdLineTask, the dataId is prepended.
- The HSC's commit on prepending the timestamps is also ported to PrependedFormatter.
- An example log message: