Details
-
Type:
Bug
-
Status: Invalid
-
Resolution: Done
-
Fix Version/s: None
-
Component/s: jenkins, pipe_tasks
-
Labels:None
-
Team:Architecture
-
Urgent?:No
Description
The preceding build log messages are:
buildConfig(["doc/doxygen.conf"], ["doc/doxygen.conf.in"])
|
doxygen /build/stack/miniconda3-py38_4.9.2-4.1.0/EupsBuildDir/Linux64/pipe_tasks-g09fcba0e3a+2ad9a2ba28/pipe_tasks-g09fcba0e3a+2ad9a2ba28/doc/doxygen.conf
|
warning: Tag 'FORMULA_TRANSPARENT' at line 1593 of file 'base.inc' has become obsolete.
|
To avoid this warning please remove this line from your configuration file or upgrade it using "doxygen -u"
|
warning: Tag 'DOT_FONTNAME' at line 2318 of file 'base.inc' has become obsolete.
|
To avoid this warning please remove this line from your configuration file or upgrade it using "doxygen -u"
|
warning: Tag 'DOT_FONTSIZE' at line 2325 of file 'base.inc' has become obsolete.
|
To avoid this warning please remove this line from your configuration file or upgrade it using "doxygen -u"
|
warning: Tag 'DOT_TRANSPARENT' at line 2581 of file 'base.inc' has become obsolete.
|
To avoid this warning please remove this line from your configuration file or upgrade it using "doxygen -u"
|
running global pytest...
|
This began happening on 2022-09-16, but only on CentOS, not macOS. Exit code 137 = 128 + 9 seems to indicate that an external process killed the build, possibly the out-of-memory killer in Kubernetes (which does not exist on macOS). The only merge was DM-35939, so suspicion immediately falls there.
Attachments
Activity
On the other other hand, even nodes that were not running this job show "Processing signal 'terminated'", so it's possible this was a rolling K8s upgrade that coincidentally happened to hit all the retries. Will try replaying the release build to see if it happens again.
DM-35939 did have a successful jenkins run before the merge.
This went away (for no apparent reason) on 2022-09-17.
This hasn't recurred.
On the other hand, Google monitoring does not show any particular memory problems on the node that ran this:
The node logs only say this:
Info 2022-09-16 04:52:59.749 PDT[2022-09-16T11:52:59] [INFO] : Container running ce8a6af0cb266b1ffab89194480bb87bba7aed0795bde9c2b174e510ff23f69b /upbeat_shockleyError 2022-09-16 05:36:05.136 PDT time="2022-09-16T12:36:05.133110719Z" level=info msg="Processing signal 'terminated'"