How Every Line of Code Impacts Performance: Lessons from Optimizing Legacy Web Applications

Overview

Our company operates 15 web applications that must deliver data‑driven functionality for real‑time decision making, even under high load. The main web app is a large, multi‑service legacy system with services older than 15 years, many of which have been refactored across generations, and the original developers are often no longer on the team.

In recent years our team focused on performance optimization of these services, and this article shares the main experiences and reasons behind the actions we took.

Cognitive Change Moment

A traffic increase caused users to complain about severe performance degradation, prompting us to investigate with monitoring tools. We discovered that 90% of response time was spent acquiring a DB connection.

Although the database appeared healthy, further analysis of the connection pool showed that all pods were exhausting the pool, suggesting connections were not being released. After hours of code review, a team leader found that the pod liveness probe performed a simple DB heartbeat without releasing the connection. Adding a line to close the connection in the probe instantly stabilized performance and restored user experience.

We had performed a load test the day before, which incorrectly indicated the system could handle the expected load, leading us to a false sense of security. This taught us the importance of accurate testing.

Lesson 1: Do not rely on average latency as the sole load metric – examine tail‑latency values (50th, 90th, 95th, 99th percentiles).

When users complained about slowness, average latency remained unchanged because many fast requests skewed the average. Monitoring tail‑latency revealed the real performance issue.

Lesson 2: Invest time, tools, and personnel in performance optimization.

Maintaining high performance requires:

Load testing and realistic load scenarios.

Application performance monitoring (APM) tools such as Dynatrace, AppDynamics, or Epsagon.

Effective logging.

Log analysis platforms (ELK, Grafana, Splunk) to handle distributed logs.

Skilled personnel to operate these tools and interpret data.

For complex systems, dedicated teams (e.g., SRE) should handle performance work.

Lesson 3: Legacy systems will decay unless actively maintained.

Neglecting old systems leads to loss of knowledge, higher MTTR, and inability to respond to incidents or new requirements. Continuous familiarity through code changes is essential.

Lesson 4: Every line of code matters; a single forgotten DB‑close caused a major outage.

My Recommendations

Imagine a user whose work depends entirely on your code; each line affects their experience.

Run load tests in CI/CD for every PR or release to ensure high availability.

When performance issues arise, scrutinize every line of code, as any character could be the bottleneck.

Conclusion

This article summarizes our full set of performance‑optimization lessons, emphasizing that application performance should be the top priority, outweighing UI polish or feature flashiness.

All conclusions stem from daily optimization practice and form the foundation for successful performance work.