Cover Image for How an Obscure HTTP Header Broke a Production App (and Why CDNs Strip It)

An obscure error

I was in NYC ~~eating gelato with some friends~~ closing a million dollar web hosting deal, when I got a message from one of our customers - vamoscaribe.com - on Discord.

They had just deployed their PayloadCMS application and despite it working locally - all the relationship fields were not working in production.

Whats worse? They couldn't find any info on the obscure error they were getting in the PayloadCMS community.

So it was time to ~~finish up this gelato~~ get this deal closed, get home, and put on my debugging hat.

FYI: Payload is an open-source Next.js backend used as a Content Management System. You can treat it as any other opiniated web-framework during your read through and still get draw the same conclusion from this article.

It works on my machine

The project worked just fine on his local machine. But in production, whenever a request was made to fetch the /api/cities endpoint from his backend. He'd get a 500 error.

Graciously, he granted me access to his admin panel and upon further investigation, I noticed that other /api/* endpoints were also failing silently. They would return 200 Success status codes but the response content would contain error messages:

This was the clue I needed. The 500 error was likely a side-effect of a more fundamental problem - one which lived in how the network layer.

Step by Step Bypassing of Network Steps

At Sherpa.sh we deploy all javascript apps using best practices for fast and scalable infrastructure. The request flow for the PayloadCMS application looked like this:

Browser Request -> CDN -> Load Balancer -> K8s Ingress -> Web App

The first test I did was forward the Web App container port to my local machine. This would effectively bypass all the network request and let me communicate directly to the live web app.

Everything worked. I was right! There was a network problem.

The next thing I did was bypass the CDN layer. I create an A record on a new domain and added it to the Load balancer and k8s ingress for the app.

This worked too! The problem is at the CDN layer!

FYI: We use BunnyCDN for our CDN layer at Sherpa.sh

Comparing Apples to Apples

Because the issue was at the CDN level, I suspected a header was getting modified or dropped by the CDN provider. So I added logging to my ingress to see the request headers were coming in and compared it to the request headers being sent by the browser in the instance flowing through the CDN.

The logs were showing me the X-HTTP-Method-Override header was getting removed. Which means the CDN was stripping this header by default.

A cursory look at the PayloadCMS issues shows since 2024 we've known this header is unsupported on many different deployment platforms - including the big ones - like AWS Amplify, Cloudfront, Cloudflare, etc.

What is X-Http-Method-Override

So what is this header anways? The X-Http-Method-Override header tells the server to treat the request as if its the method in the header instead of the actual request method used. For example, you can make a POST request, but tell the server to inteprit it as a GET request.

Why would anyone need to use this header?

It was originally intended for legacy browsers, simple http clients, and some embedded devices that only support GET and POST requests. In order to interact with modern REST apis using PUT, PATCH and DELETE these limited devices are supposed to use X-Http-Method-Override

PayloadCMS uses query args in a GET request to fetch relationship data between CMS entities. They introduced the header because projects with a large number of entities would start bumping up against URL character limits of some browsers.

By moving to a POST request and sending form data the could bypass the limit, and by passing the X-Http-Method-Override header they could get the same behavior without changing a bunch of backend code (I leave it up to the reader to decide if this was a good decision or not)

Patching with Edge Scripting

Our CDN provider has no way to whitelist headers to forward, so we deployed a script at the edge in the CDN nodes that would get the value of the X-Http-Method-Override header if it was set and re-apply it to the outbound request to the LoadBalancer.

After deploying this change, along with another test, the project was working as expected. Bug Squashed.

Now... Where's the rest of my gelato?

Takeaways

Frameworks and their developers should stick to the most common request headers whenever possible in order to maximize compatibility with infrastructure providers. Just because a header is supported by a web spec, that doesn't mean the rest of the providers on the internet are going to support it.

This isn't the first time I've seen header issues with Javascript frameworks. Next.js had a notorious bug where the stale-while-revalidate header was using the wrong format, and if you didn't account for it yourself, your Nextjs caching would not work properly.

How an Obscure HTTP Header Broke a Production App (and Why CDNs Strip It)