# The 6 MB Lambda limit: compress before, not after

> Why API Gateway compression can make large responses look safe in testing, then still fail when Lambda enforces the raw payload limit first.

- Category: Engineering
- Published: 2025-09-17
- Authors: Rohit Garg
- Canonical: https://videodb.io/blog/lambda-compression-trap
- HTML: https://videodb.io/blog/lambda-compression-trap · Markdown: https://videodb.io/blog/lambda-compression-trap.md
- Tags: aws, api, compression

---
## What we saw
We had an API endpoint whose JSON response size grew with the amount of data requested. Since it ran behind AWS Lambda, we knew unbounded responses were unsafe. Lambda has a hard response payload limit, so pagination was already the right direction.

The confusing part showed up while testing that pagination work. Responses we expected to be close to the limit looked much smaller in Postman. Then a slightly larger response from the same endpoint failed completely with `502 Bad Gateway` or `413 Payload Too Large`.

It felt like the limit was moving. A response we expected to be around 5.9 MB could appear as a tiny transfer in the client, while a response around 6.1 MB failed before we saw a body at all. The mistake was comparing the compressed size Postman showed us with the raw size Lambda was enforcing.

## Where the 6 MB limit sits
Lambda does not care how small the response might become after a proxy compresses it. It has to receive the response from the application first. If the response crossing that boundary is too large, Lambda rejects it.

In our stack, the Flask app returned JSON through Lambda. At this point the body was still raw JSON because we had not enabled app-level compression. Lambda was not shrinking it. Lambda was only enforcing the response payload limit.

That is the first byte count to keep in your head: raw bytes produced by the app and handed back through Lambda. This is the byte count that can trigger `502` or `413` before the client receives anything useful.

## How compression gets negotiated
HTTP compression is negotiated with headers. A client sends `Accept-Encoding` to say what it supports, such as `gzip`, `deflate`, or `br`. The server answers with `Content-Encoding` when it actually compresses the response.

Postman sends `Accept-Encoding: gzip` by default. Python `requests` also sends compression headers by default, so SDKs built on it usually ask for compressed responses unless they override those headers. That means normal clients may already be asking the gateway for gzip.

API Gateway CAN honor that request and compress the response, BUT ONLY AFTER Lambda has returned successfully. That ordering is the whole trap.

```text
Client:      Accept-Encoding: gzip, deflate, br
App:         returns raw JSON unless app-level compression is enabled
Lambda:      enforces raw response payload limit
API Gateway: compresses only after Lambda returns successfully
Client:      receives compressed bytes and decompresses automatically
```

<strong>Flask app</strong><span>returns JSON</span>-&gt;<strong>Lambda</strong><span>checks raw bytes</span>-&gt;<strong>API Gateway</strong><span>gzip, if allowed</span>-&gt;<strong>Client</strong><span>sees transferred bytes</span>

## Why local tests did not catch it
Once pagination was in progress, this created a misleading test result. A near-limit response could succeed, then API Gateway compressed it, and Postman showed a much smaller transfer size. A slightly larger raw response failed before API Gateway ever got a chance to compress it.

That ordering creates the split behavior we saw in pagination testing.

<p class="case-label">Raw response below limit</p><strong>~5.9 MB raw</strong><span>Lambda returns successfully</span><span>API Gateway compresses</span><span class="case-outcome">⚡ Postman shows ~100 KB transferred</span><p class="case-label">Raw response above limit</p><strong>~6.1 MB raw</strong><span>Lambda rejects the response</span><span>API Gateway never compresses</span><span class="case-outcome">🤔 Client sees 502 or 413</span>

That is what made the bug feel strange. Before hitting the limit, the response suddenly looked dramatically smaller because we were seeing compressed transfer bytes. After adding only a little more data, the request crossed Lambda's raw 6 MB boundary and broke before compression could happen. It felt like the limit was moving, but we were really switching between two different byte counts.

> Measure the bytes at the boundary that enforces the limit, not only the bytes your client reports after compression.

## What we changed
Pagination stayed the durable fix. The endpoint needed bounded pages or cursors so each response stayed safely below the raw Lambda limit. Compression is useful, but it should not become permission to return unbounded JSON.

We also added app-level compression with `flask-compress`. The important change was not "more compression" in the abstract. It was moving compression earlier, before the response crossed the Lambda boundary.

```python
from flask import Flask
from flask_compress import Compress

app = Flask(__name__)
Compress(app)

@app.get("/api/results")
def results():
    return {"items": build_large_json_response()}
```

Gateway compression can still be useful for transfer size and latency on successful responses. It just cannot protect Lambda from the raw response size limit, because it runs too late in the path.

<strong>Best default</strong><span>Paginate or return a cursor so each response is bounded.</span><strong>Tactical layer</strong><span>Compress in the app when compressible JSON still needs to be returned directly.</span><strong>For very large results</strong><span>Move to async export, object storage, or a streaming-style workflow.</span>

## What it did to the numbers
On a representative JSON response around 2.65 MB raw, compression changed both transfer size and latency. The exact numbers will vary by payload shape and network path, but the direction is the useful part.

| Setup | Request | Latency | Transferred |
|---|---|---:|---:|
| Gateway compression only | No Accept-Encoding | 2.7s | 2.65 MB |
| Gateway compression only | With Accept-Encoding | 2.0s | 330 KB |
| App plus gateway compression | No Accept-Encoding | 2.7s | 2.65 MB |
| App plus gateway compression | With Accept-Encoding | 1.8s | 281 KB |

The 2.65 MB to 330 KB case is roughly an 87 percent transfer-size reduction. The important distinction is not just 330 KB versus 281 KB. It is where the compression occurs. App-level compression can shrink compressible JSON before the response crosses the Lambda boundary. Gateway compression cannot rescue a response Lambda already rejected.

<p class="case-label">Debug these separately</p><ul><li>Raw response bytes produced by the application.</li><li>Transferred bytes after <code>Content-Encoding</code>.</li><li>Whether the request sent <code>Accept-Encoding</code>.</li><li>Whether API Gateway or the app applied compression.</li></ul>

## What we would tell another team
When a serverless endpoint returns large JSON, measure both raw bytes and transferred bytes. Browser tools, Postman, and HTTP clients often show the compressed transfer size, which can make a response look smaller than the payload your runtime actually had to return.

A good checklist is simple: know whether clients send `Accept-Encoding`, know the raw payload size, know where compression is applied, keep Lambda responses below the raw limit, paginate large results, and use app-level compression when returning compressible data directly. If the response can keep growing, move it out of the synchronous response path instead of tuning around a limit.

## Cite this work

Rohit Garg, "The 6 MB Lambda limit: compress before, not after", VideoDB Labs, September 2025.

```bibtex
@article{garg2025lambda,
  author = {Rohit Garg},
  title = {The 6 MB Lambda limit: compress before, not after},
  journal = {VideoDB Labs},
  year = {2025},
  month = {sep},
  note = {https://videodb.io/blog/lambda-compression-trap},
}
```