I'm running the httpd24u package from IUS on two Centos 7 servers. The version identifier for the package is "2.4.52-1.el7.ius".
For the last week or so, we've been seeing intermittent broken HTTP responses from the servers. The browser reports "net::ERR_CONNECTION_CLOSED", and the server error log includes output like:
*** Error in `/usr/sbin/httpd': free(): invalid next size (fast): 0x00007fb394015f10 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x81329)[0x7fb3cf231329]
/lib64/libcrypto.so.10(CRYPTO_free+0x1d)[0x7fb3c7b1396d]
/lib64/libcrypto.so.10(EVP_MD_CTX_cleanup+0xe9)[0x7fb3c7bd0a69]
/lib64/libcrypto.so.10(+0x1275ee)[0x7fb3c7bcc5ee]
/lib64/libssl.so.10(ssl23_accept+0x39)[0x7fb3c7f37409]
/etc/httpd/modules/mod_ssl.so(+0x18352)[0x7fb3c0122352]
/etc/httpd/modules/mod_ssl.so(+0x19ed6)[0x7fb3c0123ed6]
/etc/httpd/modules/mod_ssl.so(+0xdad0)[0x7fb3c0117ad0]
/usr/sbin/httpd(ap_run_process_connection+0x40)[0x55a2b89abe10]
/etc/httpd/modules/mod_mpm_event.so(+0x714a)[0x7fb3c25db14a]
/etc/httpd/modules/mod_mpm_event.so(+0x7d9e)[0x7fb3c25dbd9e]
/lib64/libpthread.so.0(+0x7ea5)[0x7fb3cf789ea5]
/lib64/libc.so.6(clone+0x6d)[0x7fb3cf2aeb0d]
======= Memory map: ========
55a2b8955000-55a2b89e1000 r-xp 00000000 08:03 1879095252 /usr/sbin/httpd
55a2b8be0000-55a2b8be3000 r--p 0008b000 08:03 1879095252 /usr/sbin/httpd
55a2b8be3000-55a2b8be5000 rw-p 0008e000 08:03 1879095252 /usr/sbin/httpd
55a2b8be5000-55a2b8be9000 rw-p 00000000 00:00 0
55a2b9fea000-55a2ba3a0000 rw-p 00000000 00:00 0 [heap]
55a2ba3a0000-55a2ba46c000 rw-p 00000000 00:00 0 [heap]
7fb34c000000-7fb34c021000 rw-p 00000000 00:00 0
7fb34c021000-7fb350000000 ---p 00000000 00:00 0
7fb354000000-7fb354074000 rw-p 00000000 00:00 0
7fb354074000-7fb358000000 ---p 00000000 00:00 0
7fb358000000-7fb358021000 rw-p 00000000 00:00 0
(The memory map continues for several pages.)
I don't know how to interpret this error message. I've learned that the "free(): invalid next size" indicates a memory error (trying to free() unallocated memory or double-free()ing the same memory).
There are a couple of possibly correlated recent events that I can think of:
- The httpd24u package updated fairly recently. All of httpd's files (modules, etc) have a last modified time of Jan 7. However, I don't think that our troubles go back that far.
- We turned on HTTP/2 recently (Feb 3). This also involved switching httpd from the "worker" to the "event" MPM, at least on one of the servers.
One obvious next troubleshooting step is to disable HTTP/2 and see if the errors go away. However, even if that helps, we don't want to run without HTTP/2 for very long because of the performance improvements it offers. What else can I try in this situation?
--
Update: Reverting to 2.4.51 (with no other changes) has improved our servers' reliability, I think because this bug was created in 2.4.52: https://bz.apache.org/bugzilla/show_bug.cgi?id=65769
