Details
Description
Python 3 buffers sys.stderr, which means error messages can hang around in the buffer until the process completes (or, if it dies, those messages will never be seen). We need to ensure that we flush the buffers after running each element.
Attachments
Activity
The change in pipe_base does an explicit flush of stdout and stderr once per data unit (e.g., of the order of minutes for processCcd.py), which should have zero performance impact.
The change in ctrl_pool ensures that the console logging to stderr is being immediately flushed, which is what one expects for stderr, so this should have no additional performance impact over what we should have (though it's not clear whether we currently have what we should have or not, so there may be a change relative to what we have now, but that is desirable and entirely the point).
Paul Price
added a comment - The change in pipe_base does an explicit flush of stdout and stderr once per data unit (e.g., of the order of minutes for processCcd.py ), which should have zero performance impact.
The change in ctrl_pool ensures that the console logging to stderr is being immediately flushed, which is what one expects for stderr , so this should have no additional performance impact over what we should have (though it's not clear whether we currently have what we should have or not, so there may be a change relative to what we have now, but that is desirable and entirely the point). Hsin-Fang Chiang, would you mind commenting on the log portions of this ticket?
Paul Price
added a comment - Hsin-Fang Chiang , would you mind commenting on the log portions of this ticket? I don't know enough about whether this change really helps, but it looks good to me and don't think it'll do harm either.
Hsin-Fang Chiang
added a comment - I don't know enough about whether this change really helps, but it looks good to me and don't think it'll do harm either. Thanks, all.
Merged to master.
Paul Price
added a comment - Thanks, all.
Merged to master. 
Also, have we measured the performance impact of this? I would think we at want stdout to remain buffered, at minimum.