A Caveat With AWS Shared Resources

Recently I’ve been releasing a new build, as usual utilizing a blue-green deployment by switching the DNS record to point to the load balancer of the previously “spare” group. But before I switched the DNS, I checked the logs of the newly launched version and noticed something strange – continuous HTTP errors from our web frameworks (Spring MVC) that a certain endpoint does not support the HTTP method.

The odd thing was – I didn’t have such an endpoint at all. I enabled further logging and it turned out that the request URL was not about my domain at all. The spare group, not yet having traffic directed at it, was receiving requests pointed at a completely different domain, which I didn’t own.

I messaged the domain owner, as well as AWS, to inform them of the issue. The domain owner said they have no idea what that is and that they don’t have any unused or forgotten AWS resources. AWS, however, responded as follows:

The ELB service scales dynamically as traffic demand changes, therefore when scaling occurs, the ELB service will take IP addresses from the AWS unused public IP address pool and assign them to the ELB nodes that are provisioned for you. The foreign domain name you see here in your case, likely belongs to another AWS customer who’s AWS resource is no longer using one of the IP addresses that your ELB node now has as it was released to the AWS unused IP pool at some stage, their web clients are very likely excessively caching DNS for these DNS names (not respecting DNS TTL), or their own DNS servers are configured with static entries and are therefore communicating with an IP address that now belongs to your ELB. The ELB adding and removing IPs from Route53 is briefly described in [Link 1] and the TTL attached to the DNS name is 60 seconds. Provided that clients respect the TTL, there should be no such issues.

I can simply ignore the traffic, but what happens if I’m in this role – after a burst my IP gets released, but some client (or some intermediate DNS resolver) has cached the information for longer than instructed. Then requests to my service, including passwords, API keys, etc. will be forwarded to someone else.

Using HTTPS might help in case of browsers, as the the certificate of the new load balancer will not match my domain, but in case of other tools that don’t perform this validation or have it cached, HTTPS won’t help, unless there’s certificate pinning implemented.

AWS say they can’t fix that at the load balancer, but they actually can, by keeping a mapping between IPs, owners and Host headers. It won’t be trivial, but it’s worth exploring in case my experience is not an exceptional scenario. Whether it’s worth fixing if HTTPS solves it – probably not.

So this is yet another reason to always use HTTPS and to force HTTPS if connection is made over HTTP. But also a reminder to not do clever client-side IP caching (let the DNS resolvers handle that) and to always verify the server certificate.

4 thoughts on “A Caveat With AWS Shared Resources”

  1. Amazon have fixed this a little since ELB first landed. Initially, an IP address stopped working very shortly after the TTL of 60 expired. Now, the TTL is still 60 but the IP address remains usable for 3600. At least, that’s what happens for ELBs in Japan. They had no choice as there was a whole mobile phone network that was forcing a minimum TTL of 1800.

    Either way, your point about HTTPS being doubly important still stands.

  2. I’ve seen the same thing in our production environment where our web server got spamed by some mobile app submitting analytics events :p

  3. The real solution is to use a CDN like CloudFront so that domain name resolution hits them instead of your ELB/ALB.

  4. Resource Groups helps you do just that. By default, the AWS Management Console is organized by AWS service. But with the Resource Groups tool, you can create a custom console that organizes and consolidates information based on your project and the resources that you use. If you manage resources in multiple regions, you can create a resource group to view resources from different regions on the same page.

Leave a Reply

Your email address will not be published. Required fields are marked *