HTTPS upgrades behind an AWS Elastic Load Balancer (ELB)

Introduction

Users of Amazon Web Services can elect to terminate SSL using an Elastic Load Balancer (ELB), rather than at the application level. This centeralizes management of certificates, and removes the burden of implementing SSL across multiple technologies in a stack.

In general, it is a good idea to force an HTTPS upgrade. This greatly enhances the integrity and privacy of users’ data in flight, and prevents against a downgrade attack. Unfortunately, ELBs do not support this functionality out of the box, instead moving responsibility for the HTTPS upgrade to the running application. This example will consider how to implement the upgrade, with a specific focus on the Apache httpd Web server.

How the Internet says to do it

Numerous blogs suggest the following Apache rewrite rules to upgrade HTTP traffic to HTTPS:

RewriteEngine On
RewriteCond %{HTTP:X-Forwarded-Proto} !https
RewriteRule (.*) https://%{SERVER_NAME}%{REQUEST_URI}

Amazon ELBs are kind enough to expose a set of X-Forwarded headers, describing the client request on the other side of the load balancer. X-Forwarded-Proto, in this case, determines the protocol through which the client requested information. In plain English, these rewrite rules read: If the client did not request an HTTPS session, perform an HTTPS upgrade. This has some downfalls:

The application is broken outside of the ELB

In the event that the X-Forwarded-Proto header is not set, the HTTPS upgrade will still be performed. This is because the header value will instead be the empty string, and ";"; != ";https";, so the condition will continue to evaluate to true.

Infrastructure as Code allows developers to write integration tests against their infrastructure. Consider the following Serverspec integration test, which ensures that the /ping endpoint used for a health check is returning successfully:

describe command('curl -L http://localhost/ping') do
        its(:stdout) { should match(/pong/) }
end

Even though -L is specified to follow redirects, this test will still fail - the HTTPS endpoint will not exist on the server. Likewise, any local development done inside a container will fail from the same reason. This undermines the use of Packer devboxes, Docker containers, and so on.

Health check endpoints could stop working

Let’s say the ELB has a health check defined at HTTP:80/ping. An ELB will execute the health check on the context of the host, not on the context of the ELB, meaning that HTTPS will be unavailable to the health check. Therefore, the health check will fail on any instances performing an HTTPS upgrade this way, leaving the application inaccessible.

A Better Way

The AWS Documentation helpfully states that there are only two possible values for the X-Forwarded-Proto header: http and https. This blog post should (hopefully) have also proven by now that the application can expect to see a third state - empty string. A better implementation of these rewrite rules, therefore, is:

RewriteEngine On
RewriteCond %{HTTP:X-Forwarded-Proto} =http
RewriteRule (.*) https://%{SERVER_NAME}%{REQUEST_URI}

The biggest issue that this implementation solves is that it will only HTTPS-upgrade traffic from behind the HTTPS-upgradable load balancer. Consider that any local testing will not have the X-Forwarded-Proto header set, bypassing the rewrite rule entirely.

What of the ELB health check? Looking at the payload sent by the ELB, the header isn’t sent for health check probes, meaning the HTTPS upgrade also will not happen for the health check:

GET /ping HTTP/1.1
host: x.x.x.x
User-Agent: ELB-HealthChecker/1.0
Accept: */*
Connection: keep-alive

HTTP/1.1 200 OK
Content-Length: 4
Content-Type: text/plain

pong

What of the integration tests? While the original test case will now pass, there is now a new requirement: an HTTPS upgrade must be made in the presence of the correct header. In keeping with IaC best practices, another test case can be added which explicitly mocks the header:

# Local traffic and the health check should not be HTTPS upgraded.
describe command('curl -L http://localhost/ping') do
        its(:stdout) { should match(/pong/) }
end

# An HTTPS upgrade should occur with X-Forwarded-Proto set to 'http'.
describe command('curl -H "X-Forwarded-Proto: http" http://localhost') do
        its(:stdout) { should match(/moved/) }
end

Conclusion

While terminating SSL at a load balancer is typically desirable, care must be taken to ensure that an application does not become dependent on the load balancer to function — the application, ideally, should be agnostic of the environment in which it is running. Enforcing an HTTPS upgrade only when possible ensures end user privacy while still enabling testing in isolation.