How a simple HTTP header mismatch led to cache poisoning

Architecture

At Motimate, a Kahoot! company, we use Cloudflare, a reverse proxy acting as a firewall and content delivery network, providing network analytics that help us better understand traffic, improve performance, and quickly respond to threats using a powerful ruleset engine. Cloudflare sits in front of a server and receives all incoming requests in the first place, passing them through a whole stack of rules responsible for monitoring security, caching, etc. Ultimately, it forwards filtered traffic to an origin server. On the server side, we use Ruby on Rails, a popular framework for building web applications. The server receives requests from Cloudflare, processes them, and returns responses to the reverse proxy that are next forwarded to clients.

Source: Cloudflare

Configuration

By default, Cloudflare caches files with certain extensions, such as .js or .css, usually static content, to improve performance by lowering the number of requests to an origin server and serving files from the closest location to a client, thereby reducing round-trip time.

Furthermore, both Cloudflare and the Motimate backend application independently enforce TLS usage, meaning all insecure requests are automatically upgraded by redirecting HTTP to HTTPS. While Cloudflare ensures a secure connection with a client, the origin server remains unaware of a client’s chosen protocol for communication with the reverse proxy and cannot assume HTTPS by default. Consequently, Cloudflare forwards the protocol to the server using an HTTP header X-Forwarded-Proto, a value controlled by Cloudflare and resistant to client tampering. The forwarded protocol is considered trusted and supposed to be used by the server.

Security Vulnerability

The backend application, developed with Ruby on Rails, accepts two HTTP headers for forwarding the protocol. One is X-Forwarded-Proto, provided by Cloudflare, and the other is X-Forwarded-Scheme. The latter takes precedence over X-Forwarded-Proto, causing the server to potentially use user-provided input to decide whether to redirect to HTTPS or maintain an already presumed secure connection.

The problematic order that prioritizes X-Forwarded-Scheme implements the rack library—the Ruby on Rails dependency—in version 2:

def forwarded_scheme
  allowed_scheme(get_header(HTTP_X_FORWARDED_SCHEME)) ||
  allowed_scheme(extract_proto_header(get_header(HTTP_X_FORWARDED_PROTO)))
end

Source: GitHub

Let’s consider an example that illustrates why the above code leads to buggy behavior. A bad actor using a web browser sends a request containing X-Forwarded-Scheme set to http under an endpoint: [GET] https://motimateapp.com/dynamic_content. Please note the URL starts with the https scheme. Cloudflare receives the request and passes it to the origin server with all the HTTP headers, both from the bad actor and those set by Cloudflare. The server receives the request from Cloudflare and, based on the X-Forwarded-Scheme header, decides to redirect to establish a secure connection (even though it’s already secure). A server response goes back to the web browser, which once again repeats the same request to [GET] https://motimateapp.com/dynamic_content, preserving all the HTTP headers, including X-Forwarded-Scheme set to http. Those back-and-forth requests and responses create a redirect loop, effectively flooding the server with unwanted traffic.

The above-described behavior doesn’t differ much from holding a refresh page button and isn’t considered a form of Denial-of-Service attack. Nevertheless, the fact that an attacker can force an unintended redirection by falsifying the protocol combined with caching server responses can become way more threatening.

So, let’s take a look at this scenario. A malicious actor sends a request including X-Forwarded-Scheme set to http under an endpoint cached by Cloudflare, for example, [GET] https://motimateapp.com/static_content.js. Cloudflare gets the request, checks if a cache is populated for it, and if not (this is a requirement to perform the attack successfully), forwards the request to the server, which redirects back with the 301 HTTP response code. Cloudflare takes the response, caches it, and returns it to the web browser. From that moment, when the client repeats the same request, it receives a response redirecting to the same endpoint straight from the Cloudflare cache. Since this cache is shared, all legitimate clients in a threat actor’s region will get the same cached redirect response, regardless of whether X-Forwarded-Scheme is provided, making the endpoint unavailable due to a redirect loop. Depending on the role of cached content in an application and the number of affected cache locations, this vulnerability can take down an entire web page. This type of attack is known as cache poisoning, as it allows malicious content to be injected into a cache shared among many, if not all, users.

Patch

Patching the described security vulnerability is straightforward. To determine a correct application layer protocol from a client by the origin server, we must use the X-Forwarded-Proto header, the trusted value the reverse proxy provides. There are many ways to fix it. One solution is to update the rack library from version 2 to version 3, prioritizing X-Forwarded-Proto over X-Forwarded-Scheme. However, if upgrading is challenging, as it may introduce breaking changes, another alternative could be to remove X-Forwarded-Scheme from request headers using Cloudflare Transform Rules. In that case, the server would only receive X-Forwarded-Proto.

Summary

The discussed vulnerability can be hard to detect and not immediately apparent. It’s crucial to remember that all service components, including communication between them, can be vulnerable. This particular one shows that the slight mismatch between what the reverse proxy assures and what the backend application accepts for forwarding the protocol may lead to a security flaw, resulting in partial or full-service disruption. It’s also worth mentioning that no CVE® record has been assigned to the issue, making it easier to slip under the radar. Unfortunately, this bug can still be exploited in the wild, prompting us to write this post and warn others.