MySQL Proxy Crashes When Closing a Connection and the PATCHES to Fix It

Some days ago, I installed a copy of MySQL Proxy that I compiled from the latest source (SVN revision 511). To my surprise, the program exited silently after a few days of use. By analyzing the core dump (see below) and digging into MySQL Proxy’s source code, I eventually located the line where the bug was in. It seems the bug could be caused when a client closes the connection before the server (backend) sending back the query result of the command COM_STMT_PREPARE. The data_free handler of command COM_STMT_PREPARE was not set up correctly. And the definition of function network_mysqld_com_stmt_prepare_result_free was missing, too.

Here’s the core dump,
# gdb mysql-proxy core.10423
GNU gdb Red Hat Linux (6.5-16.el5rh)
Copyright (C) 2006 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type “show copying” to see the conditions.
There is absolutely no warranty for GDB.  Type “show warranty” for details.
This GDB was configured as “x86_64-redhat-linux-gnu”…Using host libthread_db library
“/lib64/libthread_db.so.1″.
warning: Can’t read pathname for load map: Input/output error.
Reading symbols from /usr/local/lib/mysql-proxy/libmysql-chassis.so.0…done.
Loaded symbols for /usr/local/lib/mysql-proxy/libmysql-chassis.so.0
Reading symbols from /usr/local/lib/libgthread-2.0.so.0…done.
Loaded symbols for /usr/local/lib/libgthread-2.0.so.0
Reading symbols from /lib64/libpthread.so.0…done.
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /usr/local/lib/mysql-proxy/libmysql-proxy.so.0…done.
Loaded symbols for /usr/local/lib/mysql-proxy/libmysql-proxy.so.0
Reading symbols from /usr/local/lib/libevent-1.3e.so.1…done.
Loaded symbols for /usr/local/lib/libevent-1.3e.so.1
Reading symbols from /lib64/libnsl.so.1…done.
Loaded symbols for /lib64/libnsl.so.1
Reading symbols from /lib64/librt.so.1…done.
Loaded symbols for /lib64/librt.so.1
Reading symbols from /lib64/libresolv.so.2…done.
Loaded symbols for /lib64/libresolv.so.2
Reading symbols from /lib64/libm.so.6…done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /usr/local/lib/libgmodule-2.0.so.0…done.
Loaded symbols for /usr/local/lib/libgmodule-2.0.so.0
Reading symbols from /lib64/libdl.so.2…done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /usr/local/lib/libglib-2.0.so.0…done.
Loaded symbols for /usr/local/lib/libglib-2.0.so.0
Reading symbols from /lib64/libc.so.6…done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2…done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /usr/local/lib/mysql-proxy/libadmin.so…done.
Loaded symbols for /usr/local/lib/mysql-proxy/libadmin.so
Reading symbols from /usr/local/lib/mysql-proxy/libproxy.so…done.
Loaded symbols for /usr/local/lib/mysql-proxy/libproxy.so
warning: no loadable sections found in added symbol-file system-supplied DSO at 0×7fff5f9fd000
Core was generated by `mysql-proxy –proxy-backend-addresses=192.168.0.243:3307 –proxy-address=192.16′.
Program terminated with signal 11, Segmentation fault.
#0  IA__g_string_free (string=0×1, free_segment=1) at gstring.c:473
473           g_free (string->str);
(gdb) bt
#0  IA__g_string_free (string=0×1, free_segment=1) at gstring.c:473
#1  0×00002ab94b5bf9e6 in network_mysqld_com_init_db_result_free (udata=0×6b0830) at network-mysqld-packet.c:284
#2  0×00002ab94b5bbde3 in network_mysqld_con_free (con=0×69de60) at network-mysqld.c:307
#3  0×00002ab94b5bbf2d in network_mysqld_con_handle (event_fd=28, events=2, user_data=0×69de60) at network-mysqld.c:735
#4  0×00002ab94b7d56d9 in event_base_loop (base=0×610df0, flags=0) at event.c:331
#5  0×00002ab94b15850a in chassis_mainloop (_chas=<value optimized out>) at chassis-mainloop.c:163
#6  0×00000000004023c7 in main (argc=1, argv=0×7fff5f95cf88) at chassis.c:594

A little tip: If you run MySQL Proxy in daemon mode and it crashes, you should check the directory “/” to see whether there is a core dump file generated in it. Because MySQL Proxy chdir(2) to the root directory when daemonize()ing. Of course you should turn on core dumps (ulimit(1)) first.

Although I failed to reproduce this bug, I wrote a bug report to MySQL’s bug repository finally. I also submitted two patches to fix this problem.

The bug has been verified by Kay Roepke, one of MySQL Proxy’s developers. “Thank you for your submission. We are in the process of updating the public repository, so verifying against the current source will be easier in the future. However, this particular problem is still present in the latest version.”, Kay replied. Hmmm, so I’ve no idea when the patches would be approved and applied, but I hope we don’t wait too long to see the patches being merged into the trunk. (Oh, Jan and Kay, it seems there were no activities of MySQL Proxy quite a few months :-)

For those living on the bleeding edge, i.e. who are using the latest SVN version of MySQL Proxy and suffering from the same problem I came across, maybe you can give these patches a try, and see if it works for you. I hope this helps, though.

You can download the patches here,
network-mysqld-packet.c.patch network-mysqld.c.patch

Regards.

Updated Jan 16, 2009: latest patches for the latest MySQL Proxy (Bazaar revision 561):
network-mysqld-packet.c.patch network-mysqld.c.patch

Updated Feb 6, 2009: the patches have been merged into the trunk (Bazaar revision 566)! So the bug now has been fixed :D

Comments (2)

A bug of MySQL Proxy

I found a bug of MySQL Proxy these days, which could cause it crash. I have also located the spot where the bug hides in the source code, but I could not reproduce it. Yes, the bug lies there, as you know, but you just cannot reproduce it :-(

Anyway, I will try to reproduce this bug, and then I will submit a bug report and maybe a patch to fix it soon.

Comments (4)