Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-5176

Running runQuery.py for a long time causes mysql-proxy to crash

    Details

    • Type: Bug
    • Status: To Do
    • Resolution: Unresolved
    • Fix Version/s: None
    • Component/s: Qserv
    • Labels:
      None
    • Team:
      Data Access and Database

      Description

      mysql-proxy crashes for unknow reason:

      qserv@ccqserv125:~$ ps x
         PID TTY      STAT   TIME COMMAND
           1 ?        Ss     0:00 /bin/sh -c /qserv/scripts/start.sh
          16 ?        S      0:00 /bin/sh /qserv/scripts/start.sh
         266 ?        S      8:11 tail -F /qserv/run/var/log/mysqld.log /qserv/run/var/log/xrootd-console.log /qserv/run/var/log/worker/xrootd.log /qserv/run/var/lo
        1117 ?        Ss     0:00 bash
       14848 ?        S      0:00 /bin/sh /qserv/stack/Linux64/mariadb/10.1.11/bin/mysqld_safe --defaults-file=/qserv/run/etc/my.cnf --datadir=/qserv/data/mysql --p
       14965 ?        Sl     5:52 /qserv/stack/Linux64/mariadb/10.1.11/bin/mysqld --defaults-file=/qserv/run/etc/my.cnf --basedir=/qserv/stack/Linux64/mariadb/10.1.
       15018 ?        Sl     0:07 /qserv/stack/Linux64/xrootd/2015_11.0/bin/xrootd -c /qserv/run/etc/lsp.cf -l /qserv/run/var/log/xrootd.log -n worker -I v4
       15054 ?        Sl     0:08 /qserv/stack/Linux64/xrootd/2015_11.0/bin/cmsd -c /qserv/run/etc/lsp.cf -l /qserv/run/var/log/cmsd.log -n worker -I v4
       15152 ?        S      0:33 python /qserv/stack/Linux64/qserv/2016_01-10-gebc3522/bin/watcher.py -c /qserv/run/etc/qserv-watcher.cnf -v
       15182 ?        S      0:05 python /qserv/stack/Linux64/qserv/2016_01-10-gebc3522/bin/qservWmgr.py -c /qserv/run/etc/qserv-wmgr.cnf -v
       22132 ?        R+     0:00 ps x
      

      Here's the proxy log:

      2016-02-18 00:28:28: (critical) plugin proxy 0.8.5 started
      2016-02-18 00:28:28: (debug) max open file-descriptors = 1048576
      2016-02-18 00:28:28: (message) proxy listening on port :4040
      2016-02-18 00:28:28: (message) added read/write backend: 127.0.0.1:13306
      2016-02-18 00:29:03: (debug) [network-mysqld.c:1134]: error on server connection (fd: 130 event: 2). closing client connection.
      2016-02-18 00:32:17: (debug) proxy-plugin.c:229: connecting to 127.0.0.1:13306 timed out after 2.00 seconds. Trying another backend.
      2016-02-18 00:32:17: (critical) proxy-plugin.c.1865: Cannot connect, all backends are down.
      2016-02-18 00:32:17: (debug) last message repeated 2 times
      2016-02-18 00:32:17: (debug) [network-mysqld.c:1134]: error on a connection (fd: -1 event: 0). closing client connection.
      2016-02-18 05:48:05: (debug) last message repeated 23 times
      2016-02-18 05:48:05: (debug) [network-mysqld.c:1134]: error on a connection (fd: -1 event: 0). closing client connection.
      2016-02-18 07:29:58: (debug) last message repeated 1 times
      2016-02-18 07:29:58: (debug) [network-mysqld.c:1134]: error on a connection (fd: -1 event: 0). closing client connection.
      2016-02-18 08:28:44: (debug) last message repeated 1 times
      2016-02-18 08:28:44: (debug) [network-mysqld.c:1134]: error on a connection (fd: -1 event: 0). closing client connection.
      2016-02-18 08:31:27: (debug) last message repeated 1 times
      2016-02-18 08:31:27: (debug) [network-mysqld.c:1134]: error on server connection (fd: 396 event: 1). closing client connection.
      2016-02-18 08:31:27: (debug) [network-mysqld.c:1134]: error on server connection (fd: 380 event: 1). closing client connection.
      2016-02-18 08:31:27: (debug) [network-mysqld.c:1134]: error on server connection (fd: 374 event: 1). closing client connection.
      

        Attachments

          Issue Links

            Activity

            Hide
            salnikov Andy Salnikov added a comment -

            Is there a core file?

            Show
            salnikov Andy Salnikov added a comment - Is there a core file?
            Hide
            jammes Fabrice Jammes added a comment -

            Sorry, but I forgot to look for it, and replaced my containers which John's ones, will have to reproduce the problem using Vaikunth notes: http://dmtn-004.lsst.io/en/latest/

            FYI, runQueries.py ran successfully this morning, without sending any exceptions... I used mytop to monitor in realtime MySQL but I couldn't get any interesting information out of it.

            Show
            jammes Fabrice Jammes added a comment - Sorry, but I forgot to look for it, and replaced my containers which John's ones, will have to reproduce the problem using Vaikunth notes: http://dmtn-004.lsst.io/en/latest/ FYI, runQueries.py ran successfully this morning, without sending any exceptions... I used mytop to monitor in realtime MySQL but I couldn't get any interesting information out of it.

              People

              • Assignee:
                Unassigned
                Reporter:
                jammes Fabrice Jammes
                Watchers:
                Andy Salnikov, Fabrice Jammes, Jacek Becla
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:

                  Summary Panel