Debugging and Optimizing an Elixir API: From Redis Cache to a Regex.replace Performance Bug

Tubi’s streaming platform serves millions of users, and each video playback triggers an internal API that fetches a manifest file from AWS S3; the service’s P99 latency hovered around 400 ms, leading the team to suspect the S3 network request as the bottleneck.

After deploying a Redis cache for 50 % of traffic, the P99 unexpectedly rose by about 100 ms, prompting deeper investigation.

The first hypothesis blamed CPU overhead from compressing data with :erlang.term_to_binary/2 and :compressed before writing to Redis and decompressing with :erlang.binary_to_term/1 , but the observed 10 % CPU increase did not fully explain the latency spike.

Further analysis showed that while the P99 and average latency grew, the median (P50) actually decreased, indicating that fast requests improved but slower ones got worse.

Using eprof profiling on a production node, the team discovered that roughly 89.6 % of CPU time was spent in Regex.precompile_replacement/1 , which is invoked by Regex.replace/4 —the function responsible for updating the manifest content.

This revealed two distinct request paths: requests that did not need manifest modification were limited by S3 latency, whereas requests that did required heavy CPU work in Regex.replace/4 , and the added Redis cache increased that cost by about 100 ms.

To mitigate the issue, the engineers added a cheap guard using Regex.match?/2 to skip the replace call when no substitution was needed, cutting the latency of the slow path by up to 100 ms.

Although more elaborate parsers such as nimble_parsec were considered, the simple match‑check proved sufficient, avoiding unnecessary complexity.

Further investigation showed that Elixir’s Regex.replace/4 always pre‑compiles the replacement even when there is no match, unlike Erlang’s :re.replace which skips this step; the team submitted a PR ( #10500 ) that disables the unnecessary pre‑compilation, and the change was merged within ten minutes.

The post concludes with three lessons: test with real production data, use large‑scale benchmarks to expose CPU‑bound bottlenecks, and contribute fixes back to open‑source projects to help the broader community.