# pipe_tasks tarball builds fail with exit code 137

XMLWordPrintable

#### Details

• Type: Bug
• Status: Invalid
• Resolution: Done
• Fix Version/s: None
• Component/s:
• Labels:
None
• Team:
Architecture
• Urgent?:
No

#### Description

The preceding build log messages are:

 buildConfig(["doc/doxygen.conf"], ["doc/doxygen.conf.in"]) doxygen /build/stack/miniconda3-py38_4.9.2-4.1.0/EupsBuildDir/Linux64/pipe_tasks-g09fcba0e3a+2ad9a2ba28/pipe_tasks-g09fcba0e3a+2ad9a2ba28/doc/doxygen.conf warning: Tag 'FORMULA_TRANSPARENT' at line 1593 of file 'base.inc' has become obsolete.  To avoid this warning please remove this line from your configuration file or upgrade it using "doxygen -u" warning: Tag 'DOT_FONTNAME' at line 2318 of file 'base.inc' has become obsolete.  To avoid this warning please remove this line from your configuration file or upgrade it using "doxygen -u" warning: Tag 'DOT_FONTSIZE' at line 2325 of file 'base.inc' has become obsolete.  To avoid this warning please remove this line from your configuration file or upgrade it using "doxygen -u" warning: Tag 'DOT_TRANSPARENT' at line 2581 of file 'base.inc' has become obsolete.  To avoid this warning please remove this line from your configuration file or upgrade it using "doxygen -u" running global pytest... 

This began happening on 2022-09-16, but only on CentOS, not macOS. Exit code 137 = 128 + 9 seems to indicate that an external process killed the build, possibly the out-of-memory killer in Kubernetes (which does not exist on macOS). The only merge was DM-35939, so suspicion immediately falls there.

#### Activity

Hide
Kian-Tat Lim added a comment -

On the other hand, Google monitoring does not show any particular memory problems on the node that ran this:

The node logs only say this:

 Info 2022-09-16 04:52:59.749 PDT[2022-09-16T11:52:59] [INFO] : Container running ce8a6af0cb266b1ffab89194480bb87bba7aed0795bde9c2b174e510ff23f69b /upbeat_shockley Error 2022-09-16 05:36:05.136 PDT time="2022-09-16T12:36:05.133110719Z" level=info msg="Processing signal 'terminated'" 

Show
Kian-Tat Lim added a comment - On the other hand, Google monitoring does not show any particular memory problems on the node that ran this: The node logs only say this: Info 2022-09-16 04:52:59.749 PDT[2022-09-16T11:52:59] [INFO] : Container running ce8a6af0cb266b1ffab89194480bb87bba7aed0795bde9c2b174e510ff23f69b /upbeat_shockley Error 2022-09-16 05:36:05.136 PDT time="2022-09-16T12:36:05.133110719Z" level=info msg="Processing signal 'terminated'"
Hide
Kian-Tat Lim added a comment -

On the other other hand, even nodes that were not running this job show "Processing signal 'terminated'", so it's possible this was a rolling K8s upgrade that coincidentally happened to hit all the retries. Will try replaying the release build to see if it happens again.

Show
Kian-Tat Lim added a comment - On the other other hand, even nodes that were not running this job show "Processing signal 'terminated'", so it's possible this was a rolling K8s upgrade that coincidentally happened to hit all the retries. Will try replaying the release build to see if it happens again.
Hide
John Parejko added a comment -

DM-35939 did have a successful jenkins run before the merge.

Show
John Parejko added a comment - DM-35939 did have a successful jenkins run before the merge.
Hide
Kian-Tat Lim added a comment -

This went away (for no apparent reason) on 2022-09-17.

Show
Kian-Tat Lim added a comment - This went away (for no apparent reason) on 2022-09-17.
Hide
Kian-Tat Lim added a comment -

This hasn't recurred.

Show
Kian-Tat Lim added a comment - This hasn't recurred.

#### People

Assignee:
Kian-Tat Lim
Reporter:
Kian-Tat Lim
Watchers:
John Parejko, Kian-Tat Lim, Tim Jenness