Unknown attackers exploited a well-known weakness in the Internet’s infrastructure to divert traffic intended for Amazon servers for over two hours on Tuesday.
This attack highlighted something security experts have warned about for years. The Internet relies on a very fragile system of trust, where everyone relies on machines to broadcast the correct information about other machines. Disrupt that trust, and innocent users wind up on malicious sites without them even being aware of what happened.
The latest incident involved the Border Gateway Protocol, a protocol used by routers to communicate with other routers about the best way to reach networks. Organizations publish information about the best route to reach their networks, and other routers using BGP pick up this information. The idea is that if something goes wrong along that route, routers can publish alternate route information and the information would be disseminated in a manner of minutes so that traffic flow does not get disrupted.
"The best way to explain BGP is it’s the people in the pub who swear Dave is Dave as they’ve met him before. But sometimes they’re drunk and mistake Dave for Dennis. It’s understandable because Bob swore Dennis was Dave 12 pints in,” security researcher Kevin Beaumont said on Twitter.
Routers do not verify that the “announcements” are correct, so someone can maliciously divert users to a malicious site, as happened on Tuesday when users trying to reach their cryptocurrency wallets wound up on a spoofed site on a Russian server. The routing information can also be updated to make a company’s Internet traffic be routed through other networks before ultimately reaching the correct destination. Users may not even realize their traffic passed through different countries.
For example, in March 2011, a researcher noticed that traffic destined for Facebook on AT&T's network went through networks belonging to China Telecom and SK Broadband in South Korea before proceeding to Facebook’s own servers. While China Telecom claimed the route hijacking was a mistake and corrected the routing information in 18 minutes, many researchers have speculated that it may have been a way for China to take a look at Facebook traffic.
In 2013, Renesys noticed Internet traffic being redirected through Belarusian ISP GlobalOneBel, with traffic originating from Guadalajara, Mexico, and intended for Washington, D.C. meandering through networks in London, Moscow, Minsk, and Frankfurt, before going back to the United States.
Bigger fish to fry
In Tuesday’s attack, a range of IP addresses operated by Amazon Web Services’ DNS service known as Route 53 was mistakenly routed to a small internet service provider in Columbus, Ohio, called eNet. The ISP, also called XLHost, is connected to the Equinix fabric, and gives them access to a large number of ISPs. While Amazon was still announcing the correct routing information, XLHost was publishing a different one and other routers picked up the incorrect one. This way, traffic meant for Amazon’s DNS servers went to XLHost instead.
It’s not clear at this point how eNet picked up the fraudulent route information. Having access to eNet and the ability to manage DNS traffic requires a lot of computing power, suggesting that the cryptocurrency trading site was not the only target.
“The collateral damage victims of this attack were customers of Amazon’s Route 53 DNS Service like Instagram and CNN,” said Ameet Naik, a technical marketing manager at Internet monitoring company Thousand Eyes.
Cisco researchers say nearly 1,300 addresses got rerouted for the two-hour attack period to IP addresses associated with a Russian provider. However, this wasn’t a global attack as only 15 percent of BGP sources had picked up the invalid routing information, according to Cisco’s Doug Madory.
Executing this kind of attack isn’t easy, and neither is defending against it, experts say.
“To do this, you have to have access to the router equipment. If you can break into a router that speaks BGP, then voila,” said Barry Greene, a principal architect at Akamai. “They know exactly what they’re going after now. It’s a big Internet but it doesn’t take much to disrupt it.”
Path visualization by ThousandEyes shows how some networks using eNet/XLHost had difficulty reaching the correct Amazon IP address. (Credit: ThousandEyes)
Hard to defend against
BGP is fairly noisy, and configuration mistakes happen pretty regularly, with the correct information published within minutes. This makes it difficult to tell when announcements are malicious and when they aren’t.
For users, just the fact that they used an affected DNS resolver would have exposed them to this attack. One good thing is that the increased use of HTTPS and the fact that browsers now warn users when a certificate is signed by an unknown authority or if one is missing, means users may be able to avoid becoming victims even if they get rerouted to suspicious sites.
"If you were using HTTPS, the fake website would display a TLS certificate signed by an unknown authority (the domain listed in the certificate was correct but it was self-signed). The only way for this attack to work would be to continue and accept the wrong certificate. From that point on, everything you send would be encrypted but the attacker had the keys," wrote Cloudflare network engineer Louis Poinsignon.
Anyone who paid attention to the browser alert would have known something weird was happening and would have been able to back out. This attack highlights why it is so important to train users to heed the certificate warnings. Too many times, users are told to ignore warnings because the certificate expired or because it was a self-signed cert.
Hardening the Internet
Both BGP and the Domain Name System are critical to how traffic gets routed on the Internet, and they are both vulnerable to attack. Tuesday’s attack abused the trust-based nature of BGP to subvert Amazon’s DNS, and then abused DNS to direct users elsewhere.
There have been a number of technical proposals to secure BGP and DNS, but adoption has been slow. Dan Kaminsky has long advocated securing DNS, and DNSSEC would add a cryptographic layer to protect DNS information from being modified.
BGP hijacking is a problem because when one organization updates its routing information, other major routers update their information without verifying the change is valid. One proposed solution is to have routers verify that the IP address blocks being announced actually belong to the networks operated by that router. The Resource Public Key Infrastructure (RPKI) uses a system of cryptographic certificates that verify an IP address block belongs to the network. However, RPKI’s deployment has been slow.
Akamai’s Greene said there is some movement in the network operators’ community to start rolling out RPKI, at least in some places.
“It’s hard to deploy and that’s the challenge right now. You have to pull in people from all over the world. It’s a global problem, a world problem,” Greene said. “A layered approach is the next step. We have to figure out how we roll it out.”
Since many servers couldn't reach the correct DNS server, that meant users couldn't reach sites that relied on that DNS server. ThousandEyes analysis found that DNS errors impacted availability to Instagram during this period.