How a Single Space in ifconfig Crashed an Oracle RAC Cluster

Case Study: Oracle RAC Cluster Crash Due to Mis‑typed ifconfig Command

System operations require meticulous attention; a single misplaced space in a command can cause a disaster. This article describes an incident where a typo in the

ifconfig –a 6

command set all IP addresses to 0.0.0.0, leading to an Oracle RAC 10.2.0.4 cluster failure on Solaris 10.

Fault symptoms : Both nodes of the RAC cluster restarted around 15:00, with one instance failing to start automatically. The alert logs showed ORA‑27504 errors on both nodes.

ORA-27504: IPC error creating OSD context ORA-27300: OS system dependent operation: if_not_found failed WITH STATUS: 0 ORA-27301: OS failure message: Error 0 ORA-27302: failure occurred at: skgxpvaddr9 requested interface 192.168.168.3 NOT found. CHECK output FROM ifconfig command

The logs also recorded an IPC send timeout and a “Waiting for instances to leave: 2” message, indicating that node 2 was being expelled from the cluster because its network interface became non‑operational.

Investigation of the OS logs revealed messages such as

ip_arp_done: init failed

and high CPU usage, confirming that the network interface configuration had been altered.

Further history showed that a privileged user executed

ifconfig –a6

to check IPv6 addresses, but mistakenly typed

ifconfig –a 6

(extra space). This error caused the system to assign 0.0.0.0 to all IP addresses, breaking the heartbeat communication between the nodes.

Consequently, the cluster experienced an instant collapse, illustrating how a single space in a command can trigger a cascade of failures.

Lesson learned : Even privileged users must exercise extreme caution at the command‑level; every character matters in production environments.