How an 8‑Byte Request Size Limit Crashed Production in a Serverless Backend

In a large education‑technology project (REKT), the author was responsible for building a content‑management panel using React on the frontend and Fastify on the backend, all deployed serverlessly on AWS and managed with Terraform.

After a successful deployment and a day of testing, the team went live and began using the new feature. Later that afternoon, a customer reported that saved content was not persisting, despite the team being able to reproduce successful saves in their own tests.

The investigation involved checking logs, reproducing the issue in development, staging, and production environments, and even consulting a senior developer who helped reproduce the problem. The team discovered that requests larger than roughly 8 KB were being rejected, while smaller payloads succeeded.

Further digging revealed that the failure was not due to Fastify or the application code but was caused by an AWS Web Application Firewall (WAF) rule that limited request body size. The rule was automatically applied by the infrastructure, and changing the WAF configuration in the AWS console allowed larger payloads, temporarily fixing the issue.

However, because the infrastructure is managed by Terraform, the manual change was overwritten on the next deployment, re‑introducing the problem. The team realized they needed to update the Terraform configuration to adjust the WAF rule permanently.

The incident underscored several lessons: always verify security and networking configurations (such as WAF limits), document infrastructure constraints, and ensure that automated infrastructure code reflects required settings.

Code examples referenced in the article:

// For loop in JavaScript
for (let i = 0; i < 10; i++) {
  console.log(i);
}

# For loop in Python
for x in range(0, 10):
    print(x)

// For loop in PHP
for ($x = 0; $x <= 10; $x++) {
    echo $x;
}

By addressing the AWS WAF payload limit and updating the Terraform configuration, the team restored normal operation and highlighted the critical need for clear documentation and proactive infrastructure management.