Investigation of "Too Many Open Files" Errors in Tomcat After Apollo Config Changes

In this article, the author, a Ctrip framework R&D expert, describes a production incident where a Java web application integrated with the Apollo configuration center began reporting “Too many open files” errors after a configuration change.

The failure manifested as massive Redis connection errors and subsequent service disruption, prompting a restart of all Tomcat instances. Initial log analysis showed that only a subset of machines received the configuration update, and those machines reported NoClassDefFoundError for com.ctrip.framework.apollo.model.ConfigChange .

Investigation revealed that the root cause was the operating system’s file‑descriptor limit (Max Open Files) being set to 4096 on the affected hosts. When the configuration change triggered Tomcat’s WebappClassLoader to load new classes, it opened hundreds of JAR files simultaneously, exhausting the limit and causing both the “Too many open files” and NoClassDefFoundError exceptions.

Further experiments reproduced the issue by monitoring file‑descriptor usage with lsof and showed that after a config push the number of open JAR handles jumped from ~192 to over 400, then fell back after the loader released them.

The article also details Tomcat’s class‑loading mechanism, the periodic jar‑close thread, and provides recommendations: increase the OS file‑descriptor limits, improve monitoring and alerts for handle usage, pre‑load middleware clients at startup, and retain problematic machines for post‑mortem analysis.