Troubleshooting systemd MySQL Service Hang Caused by Premature kill -9 in Forking Mode

In an automated testing environment running MySQL 8.0.34 inside a CentOS 8 Docker container, the systemd service for mysqld sometimes hangs during startup. The service uses the forking start mode, and the ExecStart command launches /opt/mysql/base/8.0.34/bin/mysqld with appropriate options.

The symptom is a continuous hang with no output, and the MySQL error log shows no information. systemctl status reports that the service failed because the MAIN PID does not exist or is a zombie:

New main PID 31036 does not exist or is a zombie

Investigation revealed that after mysqld creates its mysqld.pid file, an automated test executes kill -9 $(cat /opt/mysql/data/11690/mysqld.pid) , terminating the process before systemd can confirm the forked child. Consequently, systemd cannot locate the PID and treats the service as failed.

Key findings:

The mysqld.pid file exists, confirming the process was started.

The process is killed by the test case before systemd finishes its ExecStartPost phase.

systemd then reports a missing MAIN PID, resulting in a zombie state.

To reproduce the issue, the service template was modified to add a sleep 10 after the ExecStart command, allowing a window to manually kill the process. The steps are:

Edit /etc/systemd/system/mysqld_11690.service and insert sleep 10 after the start command.

Run systemctl daemon-reload to apply changes.

Start the service in one SSH session and, in another session, kill the PID read from mysqld.pid as soon as the file appears.

Observe the same hanging behavior and the "New main PID ... does not exist or is a zombie" message.

The resolution is to kill the hanging systemctl start command, then issue systemctl stop mysqld_11690.service to let systemd clean up the zombie, and finally restart the service. This clears the stale MAIN PID and restores normal operation.

Although the article focuses on MySQL, the core lesson is a systematic approach to diagnosing intermittent service failures in Linux containers.