Recovering Accidentally Deleted Production Server Data Using ext3grep, extundelete, and MySQL Binlog

The author describes a production server data loss caused by a junior staff member executing a dangerous rm -rf command while trying to uninstall Oracle, which ended up deleting the entire filesystem, including Tomcat and MySQL data.

The command used was rm -rf $ORACLE_BASE/* and, because $ORACLE_BASE was unset, it became rm -rf /* , wiping the disk.

After discovering the loss, the team attempted to mount the disk on another server and inspected backups, finding only a 1 KB dump and an old backup from 2013, indicating backup failure.

As a “last‑ditch” effort, they used the ext3grep tool to scan the ext3 filesystem and list deleted files: ext3grep /dev/vgdata/LogVol00 --dump-names .

The scan produced a list of file names; they tried to restore everything with ext3grep /dev/vgdata/LogVol00 --restore-all , but insufficient space caused partial restores.

They wrote a shell script to restore only MySQL files listed in a filtered file list:

while read LINE
do
    echo "begin to restore file $LINE"
    ext3grep /dev/vgdata/LogVol00 --restore-file $LINE
    if [ $? != 0 ]; then
        echo "restore failed, exit"
    fi
done < ./mysqltbname.txt

The script recovered about 40 files, but many tables remained missing; they also tried extundelete:

extundelete /dev/vgdata/LogVol00 --restore-directory var/lib/mysql/aqsh

which failed.

Finally they remembered that MySQL binlog was enabled. They located three binlog files (mysql-binlog0001, mysql-bin.000009, mysql-bin.000010) and restored the latest one with:

mysqlbinlog /usr/mysql-bin.000010 | mysql -uroot -p

This successfully replayed transactions and recovered critical data.

The post‑mortem lists lessons: lack of proper change management, inadequate backup verification, missing monitoring, and using root for routine tasks.

The article shares useful tool links: https://code.google.com/p/ext3grep and http://extundelete.sourceforge.net .