How I Rescued a Compromised Ubuntu Server: Step‑by‑Step Intrusion Response

Background

Operating System: Ubuntu 12.04 x64

Running Services: Company business system, crawler program, data queue.

The server was hosted in an external data center. Suddenly, monitoring emails reported ping failures and traffic spiked to about 800 MB, making SSH login impossible.

1. Problem Investigation

Initially tried to cut the external network, but that would hide the traffic source. Contacted the data‑center staff and allowed them to log in. Checked

/var/log/auth.log

(already cleared) and used

iftop

to identify traffic. Found continuous HTTP traffic to IP

104.31.225.6

.

iptables -A OUTPUT -d 104.31.225.6 -j DROP

Blocking the IP temporarily reduced traffic, but it later resumed from a different IP.

2. Locating the Attack Source

Used

netstat

to filter ports, but short‑lived connections were not visible. Employed

lsof -i :15773

to find the PID, then

lsof -p PID

to inspect opened files.

netstat -atup | grep 15773

Compared file sizes with a clean system and discovered abnormal binaries (e.g.,

/bin/lsof

size 1.2 MB). Downloaded clean versions of

netstat

,

ps

, etc., and replaced the compromised tools.

3. Removing the Malware

After running

ps -ef

, several suspicious processes appeared:

pid /sbin/java.log pid /usr/bin/dpkgd/ps -ef pid /usr/bin/bsd-port/getty pid /usr/bin/.sshd

Killed and deleted these files. Noted that

java.log

reappeared, indicating a persistence script in

/etc/init.d

. Removed the script and the file, stopping the backdoor.

Incident Summary

ls /usr/bin/dpkgd/ – replaced tool, not present in a clean system. Compromised binaries: netstat, lsof, ps, ss. /sbin/java.log – packet‑sending program, regenerates after deletion. /usr/bin/bsd-port – generates java.log or other backdoor. /usr/sbin/.sshd – backdoor program.

Preventive Measures

Enable firewall after installation, allow only trusted sources, remove unnecessary users and services.

Collect logs (system, authentication, application) for early risk detection.

Monitor user logins, password retries, and command execution.

Watch critical files/directories (e.g., /etc/passwd, /etc/shadow, /web, /tmp) for changes.

Monitor process states and alert on new or suspicious processes.

Regularly scan servers and web applications for vulnerabilities.

There is no absolute security; the goal is to reduce the attack surface and improve system protection.