John Gates added a comment - 16/Mar/16 9:16 AM - edited I ran the following to put shared scan information in the cluster INSERT INTO qservCssData.kvData (kvKey, kvVal, parentKvId) VALUES ( "/DBS/LSST/TABLES/Source/sharedScan" , "" , "29" ); INSERT INTO qservCssData.kvData (kvKey, kvVal, parentKvId) VALUES ( '/DBS/LSST/TABLES/Source/sharedScan/.packed.json' , '{"lockInMem":"1","scanRating":"15"}' ,46);   INSERT INTO qservCssData.kvData (kvKey, kvVal, parentKvId) VALUES ( "/DBS/LSST/TABLES/Object/sharedScan" , "" , "33" ); INSERT INTO qservCssData.kvData (kvKey, kvVal, parentKvId) VALUES ( '/DBS/LSST/TABLES/Object/sharedScan/.packed.json' , '{"lockInMem":"1","scanRating":"5"}' ,48);   INSERT INTO qservCssData.kvData (kvKey, kvVal, parentKvId) VALUES ( "/DBS/LSST/TABLES/ForcedSource/sharedScan" , "" , "37" ); INSERT INTO qservCssData.kvData (kvKey, kvVal, parentKvId) VALUES ( '/DBS/LSST/TABLES/ForcedSource/sharedScan/.packed.json' , '{"lockInMem":"1","scanRating":"16"}' , 50);

John Gates added a comment - 17/Mar/16 11:35 AM - edited The mysql proxy becomes unresponsive while running runQueries.py and I believe this is causing connection issues. Note a variable value request takes 46 seconds shortly before runQueries fails. Increasing net_read_timeout from 30 to 60 and connect_timeout from 10 to 20 had no noticable affect. mysql> SHOW VARIABLES LIKE 'net_read_timeout'; +------------------+-------+ | Variable_name | Value | +------------------+-------+ | net_read_timeout | 60 | +------------------+-------+ 1 row in set (46.42 sec) errors (sample of errors encountered during a single runQueries.py execution, 'all backend are down' is far more frequent) File "runQueries.py", line 182, in runQueries db='LSST') File "/afs/in2p3.fr/home/j/jgates/stack/Linux64/mysqlpython/1.2.3.lsst1/lib/python/MySQL_python-1.2.3-py2.7-linux-x86_64.egg/MySQLdb/__init__.py", line 81, in Connect return Connection(*args, **kwargs) File "/afs/in2p3.fr/home/j/jgates/stack/Linux64/mysqlpython/1.2.3.lsst1/lib/python/MySQL_python-1.2.3-py2.7-linux-x86_64.egg/MySQLdb/connections.py", line 187, in __init__ super(Connection, self).__init__(*args, **kwargs2) OperationalError: (1105, '(proxy) all backends are down')     File "runQueries.py", line 182, in runQueries db='LSST') File "/afs/in2p3.fr/home/j/jgates/stack/Linux64/mysqlpython/1.2.3.lsst1/lib/python/MySQL_python-1.2.3-py2.7-linux-x86_64.egg/MySQLdb/__init__.py", line 81, in Connect return Connection(*args, **kwargs) File "/afs/in2p3.fr/home/j/jgates/stack/Linux64/mysqlpython/1.2.3.lsst1/lib/python/MySQL_python-1.2.3-py2.7-linux-x86_64.egg/MySQLdb/connections.py", line 187, in __init__ super(Connection, self).__init__(*args, **kwargs2) OperationalError: (2013, "Lost connection to MySQL server at 'reading authorization packet', system error: 0")   File "runQueries.py", line 189, in runQueries cursor.execute(q) File "/afs/in2p3.fr/home/j/jgates/stack/Linux64/mysqlpython/1.2.3.lsst1/lib/python/MySQL_python-1.2.3-py2.7-linux-x86_64.egg/MySQLdb/cursors.py", line 174, in execute self.errorhandler(self, exc, value) File "/afs/in2p3.fr/home/j/jgates/stack/Linux64/mysqlpython/1.2.3.lsst1/lib/python/MySQL_python-1.2.3-py2.7-linux-x86_64.egg/MySQLdb/connections.py", line 36, in defaulterrorhandler raise errorclass, errorvalue InterfaceError: (-1, 'error totally whack')

John Gates added a comment - 17/Mar/16 3:26 PM - edited Running queries seems to cause mysql-proxy to leak memory when queries are run, according to top output. The amount of memory leaked appears to related to the size of the results.

John Gates added a comment - 31/Mar/16 2:31 PM runQueries.py, at least the old version works again. The problem was that mysql-proxy was having to wait while the czar distributing the jobs for the user query to the worker. Putting the job distribution in a separate thread appears to have fixed the problem.

Andy Salnikov added a comment - 31/Mar/16 3:14 PM John, I have lied to you when I said yesterday that new in-process czar should behave exactly as old XML-RPC stuff. I have re-checked the old implementation and I see that it's not true, in the old code UserQuery::submit() was indeed executed in a separate thread, so czar returned earlier. In the new code we can do the same by moving call to uq->submit(); inside finalizer lambda in Czar::submitQuery() except that we cannot do it because submit() calculates queryId and result table name which are needed by czar when it returns from Czar::submitQuery() . I would suggest to move the code that registers query in QMeta from submit() into constructor, then we can execute the whole submit() method in a finalizer thread, I think that should make the code abit more cleaner than your current solution. Sorry for this mess, I should have checked it sooner

John Gates added a comment - 31/Mar/16 3:36 PM I'll need to look closer. It looked like there was a minor race condition in Executive::add(), which is fixed.

Andy Salnikov added a comment - 01/Apr/16 1:31 PM John, looks OK to me, couple of minor comments in PR.