Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-36257

pipe_tasks tarball builds fail with exit code 137

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Invalid
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: jenkins, pipe_tasks
    • Labels:
      None
    • Team:
      Architecture
    • Urgent?:
      No

      Description

      The preceding build log messages are:

      buildConfig(["doc/doxygen.conf"], ["doc/doxygen.conf.in"])
      doxygen /build/stack/miniconda3-py38_4.9.2-4.1.0/EupsBuildDir/Linux64/pipe_tasks-g09fcba0e3a+2ad9a2ba28/pipe_tasks-g09fcba0e3a+2ad9a2ba28/doc/doxygen.conf
      warning: Tag 'FORMULA_TRANSPARENT' at line 1593 of file 'base.inc' has become obsolete.
               To avoid this warning please remove this line from your configuration file or upgrade it using "doxygen -u"
      warning: Tag 'DOT_FONTNAME' at line 2318 of file 'base.inc' has become obsolete.
               To avoid this warning please remove this line from your configuration file or upgrade it using "doxygen -u"
      warning: Tag 'DOT_FONTSIZE' at line 2325 of file 'base.inc' has become obsolete.
               To avoid this warning please remove this line from your configuration file or upgrade it using "doxygen -u"
      warning: Tag 'DOT_TRANSPARENT' at line 2581 of file 'base.inc' has become obsolete.
               To avoid this warning please remove this line from your configuration file or upgrade it using "doxygen -u"
      running global pytest...
      

      This began happening on 2022-09-16, but only on CentOS, not macOS. Exit code 137 = 128 + 9 seems to indicate that an external process killed the build, possibly the out-of-memory killer in Kubernetes (which does not exist on macOS). The only merge was DM-35939, so suspicion immediately falls there.

        Attachments

          Activity

          Hide
          ktl Kian-Tat Lim added a comment -

          On the other hand, Google monitoring does not show any particular memory problems on the node that ran this:

          The node logs only say this:

          Info 2022-09-16 04:52:59.749 PDT[2022-09-16T11:52:59] [INFO] : Container running ce8a6af0cb266b1ffab89194480bb87bba7aed0795bde9c2b174e510ff23f69b /upbeat_shockley
          Error 2022-09-16 05:36:05.136 PDT time="2022-09-16T12:36:05.133110719Z" level=info msg="Processing signal 'terminated'"
          

          Show
          ktl Kian-Tat Lim added a comment - On the other hand, Google monitoring does not show any particular memory problems on the node that ran this: The node logs only say this: Info 2022-09-16 04:52:59.749 PDT[2022-09-16T11:52:59] [INFO] : Container running ce8a6af0cb266b1ffab89194480bb87bba7aed0795bde9c2b174e510ff23f69b /upbeat_shockley Error 2022-09-16 05:36:05.136 PDT time="2022-09-16T12:36:05.133110719Z" level=info msg="Processing signal 'terminated'"
          Hide
          ktl Kian-Tat Lim added a comment -

          On the other other hand, even nodes that were not running this job show "Processing signal 'terminated'", so it's possible this was a rolling K8s upgrade that coincidentally happened to hit all the retries. Will try replaying the release build to see if it happens again.

          Show
          ktl Kian-Tat Lim added a comment - On the other other hand, even nodes that were not running this job show "Processing signal 'terminated'", so it's possible this was a rolling K8s upgrade that coincidentally happened to hit all the retries. Will try replaying the release build to see if it happens again.
          Hide
          Parejkoj John Parejko added a comment -

          DM-35939 did have a successful jenkins run before the merge.

          Show
          Parejkoj John Parejko added a comment - DM-35939 did have a successful jenkins run before the merge.
          Hide
          ktl Kian-Tat Lim added a comment -

          This went away (for no apparent reason) on 2022-09-17.

          Show
          ktl Kian-Tat Lim added a comment - This went away (for no apparent reason) on 2022-09-17.
          Hide
          ktl Kian-Tat Lim added a comment -

          This hasn't recurred.

          Show
          ktl Kian-Tat Lim added a comment - This hasn't recurred.

            People

            Assignee:
            ktl Kian-Tat Lim
            Reporter:
            ktl Kian-Tat Lim
            Watchers:
            John Parejko, Kian-Tat Lim, Tim Jenness
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                Jenkins

                No builds found.