Resolving MySQL OOM Issues Caused by Full‑Text Indexes: Tcmalloc vs. Jemalloc Memory Management

1 Fault Phenomenon

A business monitoring alarm reported insufficient memory, and the mysqld process was killed by the OOM killer and automatically restarted.

[root@xxxxxx ~]# ps -ef|grep mysqld
root      17117 62542  0 20:26 pts/1    00:00:00 grep --color=auto mysqld
mysql    27799     1  7 09:54 ?        00:48:32 /usr/sbin/mysqld --daemonize --pid-file=/var/run/mysqld/mysqld.pid
[root@xxxxxx ~]#
# 操作系统日志记录 MySQL 被 OOM
Dec  8 09:54:42 xxxxxx kernel: Out of memory: Kill process 22554 (mysqld) score 934 or sacrifice child
Dec  8 09:54:42 xxxxxx kernel: Killed process 22554 (mysqld), UID 27, total-vm:11223284kB, anon-rss:7444620kB, file-rss:0kB, shmem-rss:0kB

2 Fault Analysis

The machine has 8 GB total memory, with other applications using a small portion. After MySQL restarted, memory usage remained high. The effective memory dropped sharply: at 09:55:54 it was 2 869 899 264 bytes, and at 10:00:54 it fell to 56 885 248 bytes.

2023-12-08 10:20:54    
963796992
2023-12-08 10:15:54    
93224960
2023-12-08 10:10:54    
111407104
2023-12-08 10:05:54    
113987584
2023-12-08 10:00:54    
56885248
2023-12-08 09:55:54    
2869899264

InnoDB buffer pool size is 1 GB:

| innodb_buffer_pool_size   | 1073741824     |

Using top shows MySQL currently consumes about 6 GB:

27799 mysql    20   0 8888376   6.1g   6120 S  26.2 80.5  30:19.01 mysqld

Calling malloc_stats() via gdb reveals that the Tcmalloc page‑heap freelist holds nearly 6 GB, indicating a memory‑intensive operation that does not release memory back to the OS. The suspicious operation is a full‑text query:

# Time: 2023-12-08T01:52:23.084854Z
# User@Host: xxxxxx @  [x.x.x.x]  Id: 259892877
# Query_time: 1.436714  Lock_time: 0.000049 Rows_sent: 1  Rows_examined: 0
SET timestamp=1702000343;
SELECT count(*) FROM `xx` inner JOIN (select id from xx_content  where MATCH(content) AGAINST('"Elasticsearch Cluster in 1 state"' IN BOOLEAN MODE)) al ON xx.id = al.id WHERE (xx.handle_status in ('pending','processing','completed')) AND `xx`.`sub_type` = 1;
... (additional similar queries omitted for brevity) ...

The table xx_content has a FULLTEXT index on the content column and contains 360 215 rows.

CREATE TABLE `xx_content` (
  `id` bigint(20) NOT NULL AUTO_INCREMENT,
  `content` longtext,
  PRIMARY KEY (`id`),
  FULLTEXT KEY `ngram_content` (`content`) /*!50100 WITH PARSER `ngram` */
) ENGINE=InnoDB AUTO_INCREMENT=100377976 DEFAULT CHARSET=utf8mb4;

Full‑text related variables are at their default values (e.g., ft_min_word_len=4 , ft_max_word_len=84 ).

ft_boolean_syntax       | + -><()~*:""&|
ft_max_word_len        | 84
ft_min_word_len        | 4
innodb_ft_min_token_size | 3
innodb_ft_max_token_size | 84
...

Running the same queries on a replica (which does not serve traffic) reproduces the memory increase: the mysqld process memory rises from ~3.4 GB to ~5.4 GB during query execution and does not shrink afterward.

# ps aux|grep mysqld|grep -v grep|awk '{print $6}'
3453980
... (values increase up to 5412200) ...

Conclusion: the full‑text query consumes a large amount of memory under Tcmalloc, and the allocator does not return it to the OS.

3 Solution

Switching the memory allocator from Tcmalloc to Jemalloc resolves the issue. After installing Jemalloc and setting LD_PRELOAD=/usr/lib64/libjemalloc.so.1 , MySQL releases memory after the query finishes, dropping from a peak of ~2.7 GB back to ~0.9 GB.

# yum install -y jemalloc
# cat /etc/sysconfig/mysql
#LD_PRELOAD=/usr/lib64/libtcmalloc.so
LD_PRELOAD=/usr/lib64/libjemalloc.so.1
# systemctl restart mysqld

# while true;do ps aux|grep mysqld|grep -v grep|awk '{print $6}';sleep 1;done;
822948
... (memory rises to 2738040 then falls back to 916400) ...

4 Summary

In production, MySQL typically runs with Tcmalloc and remains stable, but the rare use of full‑text indexes can trigger OOM. Replacing Tcmalloc with Jemalloc controls memory consumption and prevents similar outages.