A bombshell report from Bloomberg BusinessWeek alleged the Chinese government implanted tiny microchips into servers used by 30 organizations, including defense contractors, banks, and other large organizations, to gain “a stealth doorway into any network that included the altered machines.” According to Bloomberg, these chips would give attackers full controll over the system, letting them perform various activities remotely, such as installing malware, modifying the operating system.
Security experts have long warned about potential supply chain attacks on hardware, especially since manufacturers source chips and components from different suppliers. It would be hard for someone to look at a given circuit board with all the different types of chips and detect the one small piece that should not be there. From that perspective, Bloomberg’s claim that contractors in China modified Super Micro server motherboards by inserting tiny microchips “not much bigger than a grain of rice” after they were approached by operatives from an People’s Liberation Army unit specializing in hardware attacks, is plausible.
“In some cases, plant managers were approached by people who claimed to represent Supermicro or who held positions suggesting a connection to the government. The middlemen would request changes to the motherboards’ original designs, initially offering bribes in conjunction with their unusual requests. If that didn’t work, they threatened factory managers with inspections that could shut down their plants. Once arrangements were in place, the middlemen would organize delivery of the chips to the factories,” the Bloomberg article said.
If Bloomberg’s report is correct, this would be the clearest—and most audacious—example of how Chinese spies have subverted the manufacturing process for hardware used by U.S. companies to create a hard-to-detect long-term backdoor into those networks. As the reporters wrote, the discovery of these servers sent “a shudder through the intelligence community” because compromised servers could potentially be found in Department of Defense data centers and other military networks.
Flat-out Denials
Except Amazon and Apple, the two companies Bloomberg specifically named as being among the impacted organizations, have emphatically denied discovering and removing compromised servers in their infrastructure or notifying the U.S. authorities. Their strong denials are unusual because they specifically addressed the points raised in the article and didn’t play wordgames.
Amazon Web Services CISO Steve Schmidt didn’t mince words in his statement: “At no time, past or present, have we ever found any issues relating to modified hardware or malicious chips in SuperMicro motherboards in any Elemental or Amazon systems.” Schmidt also denied “that AWS knew about servers containing malicious chips or modifications in data centers based in China, or that AWS worked with the FBI to investigate or provide data about malicious hardware.”
[Apple](https://www.apple.com/newsroom/2018/10/what-businessweek-got-wrong-about-apple/" was equally adamant: “On this we can be very clear: Apple has never found malicious chips, ‘hardware manipulations’ or vulnerabilities purposely planted in any server.” The company also said it has “conducted rigorous internal investigations” based on Bloomberg’s inquiries and “each time we have found absolutely no evidence to support any of them.”
At first glance, there is no reason to doubt Bloomberg’s explosive findings. Even though the article itself did not have a lot of technical details, a lot of what it said matches what security experts have said how these attacks would work. A rogue chip placed between the baseboard management controller (BMC) and its SPI flash or serial EEPROM storage could intercept the signals when the BMC fetched and executed code from these locations. The chip would be able to inject malicious code and modify BMC's firmware. The BMC gives administrators full control over the box for monitoring and repair so attackers would be able to modify the operating system or connect to the server through the compromised BMC.
“There’s nothing technically infeasible about what’s described,” said Joe Fitzpatrick, a hardware security researcher and trainer, but the lack of technical details raised a lot of questions. “My whole business is teaching hardware threats and I'm skeptical every time I see claims about them.”
Why Bother with Hardware?
BMCs are notoriously old, unmatched embedded Linux systems with deep access to different parts of the server. It’s trivial to modify the firmware of most BMCs and trivial to exploit, and there are plenty of places on the circuit board where the extra component could be placed to intercept chip-based instructions. While a hardware implant-style attack is possible, a bigger—and more likely—risk would be software and network attacks against the BMC.
“Most of the capabilities of the described attack could be implemented in BMC firmware, be just as stealthy, and be far less costly to design and implement,” FitzPatrick said on Twitter. That Twitter thread also referenced work by Eclypsium, a company focused on protecting firmware, which found that BMC firmware is not signed on many Super Micro models, so software implants would work just as well.
Supply chain attacks targeting the hardware are rare because the amount of coordination required to pull of such an attack is extensive. The adversary would need to have deep understanding of the design and how different components interact in order to craft implants that small. There is an influence operation, either by coercion or bribes, to get the manufacturer to deviate from the original designs to insert the implants. This kind of attack would not want the chip added to every server as that would increase the chance of detection. So there would have to be careful targeting to make sure the modified servers went to the correct victim organization. This kind of surveillance operation would require millions of dollars and years of work to succeed.
“Technically it is feasible and we know there are weaknesses in the supply chain and we know the NSA has done interdiction. The level of effort to make it happen seems above and beyond what it’s worth because you could do the same thing in firmware,” said Joe Grand, hardware expert and founder of Grand Idea Studio.
For example, it would be far cheaper to modify the firmware of an existing chip rather than try to create a new one.
Can’t Ignore the Possibility
Even though there are a lot of questions and none of Bloomberg’s sources went on the record, the fact remains that many people confirmed various parts of the report. It’s likely there will be more revelations down the road to fill in the gaps.
For enterprises, this is a valuable lesson because the report clearly illustrates the supply chain dangers security experts have been warning about. Hardware attacks are enticing because the payoffs can be tremendous, and there are many ways for adversaries to make sure the motherboards are modified before they leave the manufacturer.
“It’s only time before capabilities like this are better understood, and more regularly used, and within the reach of a much larger range of attackers,” FitzPatrick said.
More worryingly, detecting this kind of tampering is extremely difficult because no one knows what to look for. With malware detection, there are signatures based off known-samples, but there is no such thing as a signature for hardware implants. The Bloomberg report claimed that one version of the rogue chip was sandwiched between the fiberglass layers of the PCB. Using an X-ray would indicate something on the board, but it wouldn’t be able to say what that something is, or whether it is malicious. A likely detection method is network traffic—but it would require a lot of extra work to determine that a chip is the source of the traffic and not software.
“We don’t even know what a modified system looks like. It could be a tiny chip, hidden under another chip,” said Grand. “Anybody who has these devices in their environment would have a really hard time. Without concrete details there's nothing they can do.”
Evidence suggests this was a very well-planned surveillance operation, which makes it even more likely that it wasn’t added to every Super Micro server. Just because an organization bought a Super Micro server doesn’t meant they are likely to have one with the compromised motherboard. The victim organizations were carefully selected in order to find the correct designs, and even then, every server based on those designs were likely not modified. Bloomberg described the level of targeting and coordination required in this kind of an attack as “a feat akin to throwing a stick in the Yangtze River upstream from Shanghai and ensure that it washes ashore in Seattle.”
“If the supply chain compromise capabilities are even remotely like what is being reported, there will be more vendors than Supermicro affected,” Matt Blaze, associate professor of computer and information science at the University of Pennsylvania, wrote on Twitter. “They just happened to be the first one to have their boards scrutinized this heavily.”
Just having a Super Micro server doesn’t mean it has the rogue chip. And even if the server had a backdoor component, that doesn’t mean the spies were targeting that organization.
“Finding out for sure is nearly impossible,” FitzPatrick said. “If you have a state-actor hardware attacker, you need state-actor-caliber hardware detection to notice it.” Fitzpatrick said.