Deep Decoding of China's Chip Spy Scandal: Who is Lying?

Source: Content From "theregister", Author: Kieren McCarthy, translator: @花总失了金箍棒, thank you.

A bombshell report from Bloomberg today claimed that Chinese government agents successfully implanted spy chips into servers made by Super Micro and used by Amazon, Apple, the US government and a few other potential customers, allowing Beijing to snoop on highly sensitive data.

The feature took a year to dig into an incident that allegedly occurred three years ago and had a huge impact on the market: the companies at the center of the storm - AMD, based in San Jose, has seen its stock price fall nearly 50%; accordingly, Apple's stock price has also fallen two points, and Amazon's has fallen more than two points.

However, the article was met with strong denials from the three main companies involved: Apple, Amazon, and AMD. Each company issued a strong and seemingly unequivocal statement denying that such chips existed or that any U.S. intelligence agencies had investigated the implants.

These statements are reviewed by lawyers to ensure that these listed companies will not be sued and compensated for disclosing false information. Similarly, Bloomberg employs a team of senior reporters and editors who check and improve the stories and have zero tolerance for false reports.

So which is true: that the Chinese government has successfully infiltrated the hardware supply chain and installed spy chips in highly sensitive American systems, or that Bloomberg reporters are simply imagining things and going off the track? We’ll be digging deeper.

First, the key detail disclosed in this report is that microchips that look like signal conditioning couplers are implanted on the motherboards of Advanced Micro Devices' data center servers, which are manufactured by a Chinese subcontractor.

Allegedly, the spy chips were not part of the original motherboard design but were secretly added after the factory owners were coerced or bribed to change the blueprints. We were told that the surveillance chips contained enough memory and processing power to effectively create a (hardware) backdoor in the host system, allowing outside agents to, for example, penetrate a server and steal information.

The Bloomberg article doesn't go into much technical detail, so most of us have to guess how these hacks work. From what we can tell, the spy chip is designed to look like an insignificant component on a motherboard, with a few connector pins on it - just enough for power and a serial interface. One version is said to be sandwiched between layers of fiberglass on a printed circuit board.

The spy chip can be placed between the baseboard management controller (BMC) and its SPI flash or serial EEPROM, which contains the BMC's firmware. So when the BMC extracts and executes code from memory, the spy chip will intercept the signal and modify the bitstream, injecting malicious code into the BMC processor, allowing it to seize control of the BMC.

The BMC is a key component on the server motherboard. It allows administrators to remotely monitor and repair the server without having to find it in the data center's computer room, remove it from the rack, fix it, and redeploy it. The BMC and its firmware can reboot the server, reinstall or modify the operating system, mount secondary storage containing malicious code and data, access virtual keyboards and terminals connected to the server, and more. If you can penetrate the BMC and its software, you have complete control over the server.

With a BMC compromised, a spy could potentially modify the controller firmware and/or the host operating system and software, allowing an attacker to gain access and steal data. We have been reporting on BMC security issues for some time.

Here’s a layman’s description of how the spy chip works from Bloomberg: The components involved “manipulate the core operating instructions that tell the server what to do as data passes through the (bus) on the motherboard… This all happens at a critical moment, taking advantage of the operating system’s cache mechanism (the CPU reads data directly from the motherboard’s cache). The hardware is implanted on the motherboard in a way that effectively edits the queue of information, injecting code or changing the order of instructions that the CPU follows.”

There are a few things to keep in mind: first, abnormal network traffic on the compromised server should be detectable; second, although unconventional, it is not impossible to compromise the host system by tampering with the BMC firmware, and various methods are described here (link here).

"It's technically sound," Jack Williams, a senior information security expert with the U.S. military, said during a hastily organized online conference Thursday morning. "If I wanted to do it, I would do it."

The BMC would be a "good place" to put a spy chip, Williams said, because the controller has access to the server's main memory, allowing it to inject backdoor code into the host operating system kernel. From there, it could download further spyware and execute it if no firewall rules were set up to do so.

The third thing to consider is this: if the whole thing is true, a lot of effort went into this surveillance operation. This isn't something that was casually added to Supermicro servers shipped to regular buyers - the targeting had to be extremely precise in order to avoid detection. If you bought a kit from Supermicro, we'd guess it's unlikely that there was a spy chip like this in there, if the report is true.

Fourth point: Why go to all the trouble of secretly installing another chip on the motherboard, when you can bribe and pressure the manufacturer to do it, and just replace an existing chip on the integrated circuit board? Why not replace the (original) SPI flash memory with a backdoored component - one that looks exactly like the original chip? Maybe a disguised signal coupler is the way to go.

Point 5: The chip is said to be as thin as a pencil lead. It is not impossible to use it to intercept and rewrite data from SPI flash or serial EEPROM. But it must store enough data to replace the BMC firmware code and then change the running operating system or otherwise open a backdoor. Either the Bloomberg article incorrectly describes the chip, or the (rice-sized chip) is just a schematic and the real device is much larger, or the most advanced customized semiconductor manufacturing process is used here.

Finally: You would expect companies like Apple and Amazon to have systems in place to detect not only unusual network traffic, but unusual operating system states. It makes sense to set alerts on any changes to the kernel and software stack during or after OS boot.

Bloomberg claims that the chip was first discovered in 2015 during a third-party security audit of servers manufactured by AMD, when a company called Element Technologies was undergoing due diligence before a merger. Element Technologies' servers manufactured by AMD are generally used for ultra-high-speed video processing.

Amazon reported its findings to authorities, according to Bloomberg, a report that sent shockwaves through the U.S. intelligence community because similar motherboards are used in Defense Department data centers, CIA drones and Navy shipboard networks.

Around the same time, Apple also discovered the chiplets, and according to the report, "strange network activity and firmware issues were detected." Apple contacted the FBI and gave the agency access to the suspicious hardware. Subsequently, U.S. intelligence agencies traced the source of the hardware through the supply chain and screened intercepted communications using various monitoring programs, eventually focusing on four subcontracted factories in China.

According to Bloomberg, U.S. intelligence agencies then discovered how the implantation process worked: “Managers at the factory were approached by people claiming to represent AMD or implying that their positions were connected to the government. These middlemen demanded changes to the original design of the motherboards, and they initially offered bribes for these unusual requests. When that didn’t work, they threatened managers to conduct inspections of the factory, which could result in the factory being shut down. Once a compromise was reached, the middlemen arranged for the chips to be shipped to the factory.”

This explanation seems plausible: it fits with what we know about the investigative methods of U.S. intelligence agencies, their espionage programs, and the way the Chinese government works when it interacts with private industry.

The report also provides various circumstantial evidence to add weight to its story, namely the subsequent actions of Apple and Amazon. In just a few weeks, Apple completely kicked Super Micro out of the supplier list, even though they had originally planned to purchase thousands of motherboard OEM orders. Amazon sold its Beijing data center to its local partner, Sinnet, for $300 million.

Both of these events happened at a fortuitous time, if they were the direct result of the investigation. But Apple claims that it abandoned Supermicro because it found malware on the servers that Supermicro provided to store consumer data: In 2015, a downloadable network interface driver was infected and accidentally installed on Apple's internal development equipment. There was another problem with the network cards on the server motherboards: They used outdated firmware that included a known security vulnerability.

Amazon said the sale to Sinnet was made to comply with "new Chinese regulations on equity ratios for overseas cloud service providers to continue operating in China" and had nothing to do with the discovery of the spy chip.

So far, you can trust Bloomberg's story and dismiss Amazon, Apple, and AMD's refusal to acknowledge an understandable secret national security investigation and attempt to cover up the scandal.

The denials are more explicit and specific than the typical noncommittal "denials." Even in the current political environment, public companies are unlikely to tell outright lies because if they are found to be deceiving investors, the market and regulatory consequences cannot be ignored. Usually, evaluating whether a company is telling the truth involves not only carefully analyzing their statements, but also looking at which aspects of the story they are avoiding.

Typical blame-shifting tactics are over-statements, the use of emotional generalities, or a focus on overly specific aspects — thus obscuring key aspects of the allegation — or unnecessary evasions — which make the denial sound specious.

There are examples of this in the statements issued by the above companies. For example, Amazon mentioned some old rumors in its response to Bloomberg, "There are too many falsehoods in the part involving Amazon in this article to count." This is a typical perfunctory approach that not only does not help solve practical problems, but also arouses suspicion.

It also called the story that it was selling its Beijing data center to get rid of the contaminated servers "ridiculous" — a strong emotional term that, if the story was true, would not be a ridiculous decision at all.

But Amazon also said: "It is untrue that AWS (Amazon Cloud Service) knew about supply chain compromises (under pressure), malicious chips, and other issues, and that the hardware was modified when it acquired Element Technology. It is also untrue that AWS knew that servers located in Chinese data centers contained malicious chips or were modified, or that AWS cooperated with the FBI to investigate or provide it with malicious hardware data."

You can weigh this statement. For example, the key element in the first denial is “at the time of the acquisition of Element Technologies.” How exactly is this time frame defined? How do you define “AWS”? Are the security personnel who made the relevant decisions from AWS, or from somewhere else in Amazon?

If Amazon wanted to outright deny the story, it could have said something like this: "AWS and Amazon deny Bloomberg's assertions. We are not aware of any supply chain compromises, malicious chips, or hardware modifications related to Elemental Technologies or Super Micro."

In the second denial, the language got even more drastic: "At no time. In the past or now have we been aware of hardware modifications or malicious chips on any Supermicro motherboards in any Elemental Technologies or Amazon systems. We are not involved in the government's investigation."

This is a stronger statement that is hard to fault. The denial seems pretty straightforward. But it still leaves room - the "we" in the wording - "at no time did we find out." Strictly speaking, it was not Amazon that was responsible for the security review at the time, but a third party company. At this point, things become a little tricky.

Amazon further denied other claims related to the Super Micro circuit board - which seems to mean that Bloomberg's conclusion is untenable. But the other problems with the circuit board are not enough to overturn the spy chip explanation, and in fact it is still possible for a third party to install anything they want on the motherboard through such a chip.

Apple's denial is still typical of Apple. Reflecting a sense of superiority, it mocked the news organization: "Over the past year, Bloomberg has contacted us many times about the so-called Apple security incident, sometimes vaguely, sometimes deliberately. Each time, we conducted a rigorous internal investigation, and according to the relevant investigations, each time we found no evidence to support any of the claims."

In the statement, it also talked about its "deep disappointment" with the reporters because they "were not open enough to exploring the possibility that they or their sources might be wrong or misled." It even suggested that they might have "conflated an earlier incident that occurred in 2016, where an infected driver was discovered on a Supermicro server at an Apple lab."

So far, so Apple. But it also makes one strong denial worth noting: "We can be very clear at this point: Apple has never been aware of malicious chips, 'hardware manipulation,' or intentionally planted vulnerabilities in any of its servers. Apple has never been in contact with the FBI or any other agency regarding this matter. We are not aware of any FBI investigation or any engagement with law enforcement."

It’s a strong denial no matter how you slice it, and even if the Bloomberg story turns out to be true, it’s hard to paint it as a lie.

It’s also worth noting that neither Amazon nor Apple responded with their usual “it’s our policy not to discuss any national security or law enforcement issues” refrain — the most common no-comment acquiescence.

As for AMD, it denied that it knew anything about the investigation—which is probably entirely true—but again, it doesn’t affect the story. No one has testified that AMD knew specifically that its products had been tampered with. The server maker ultimately “strongly refuted reports that server motherboards it sold to customers contained malicious chips.”

Um, a invited cybersecurity wizard?

So let’s simply switch to a different tack: Where did this story come from, who was Bloomberg’s source, and where does it go wrong?

If you look closely at this story, the most likely starting point for the investigation is a meeting organized by the Pentagon in late 2015. The report describes it as "a small, closed-door meeting in McLean, Virginia," attended by a group of technology executives and investors.

The fact that the meeting took place in McLean, near CIA headquarters rather than in a more formal setting, suggests that it was an informal gathering, and the number of people present made it easy for participants to leak details to reporters without exposing themselves.

The meeting came shortly after President Obama reached a cybersecurity agreement with his Chinese counterpart, in which China said it would no longer ignore the theft of intellectual property from American companies. According to Bloomberg's sources, some in the intelligence community are concerned that China has developed more advanced ways to hack into servers - the report suggests that the next generation of spy chips may be thin enough to be embedded in the fiberglass panels that assemble other components.

The central detail of the story — that U.S. intelligence agencies investigated after being informed by the private sector of the possible presence of spy chips on server motherboards — can be traced back to this meeting.

Bloomberg's version of the meeting description said, "Attendees were not told the name of the hardware maker involved, but it was clear to at least some in the room that it was from Super Micro."

Given this lead, Bloomberg reporters have been pursuing the story, and from what we know, two other key sources - someone who claims to have seen confidential internal reports from Amazon and its third-party contractors that included this lead, and a second person who has "seen the digital photos and X-rays of the chips."

Bloomberg said the third-party (security review) contractor is based in Ontario, Canada. Amazon insisted that it "commissioned an external security company to conduct a security assessment for us, and the report did not find any problems with the modified chips or hardware." It reiterated this: "This is the only external security report (Amazon) commissioned," and pointed out that Bloomberg "refused to share any details of the alleged other reports with us."

This makes you wonder: Where did this alleged report come from? Who commissioned it? Who wrote it? And are we supposed to believe anyone saw it? The whole story hinges on Bloomberg’s claim that this report exists, and Amazon’s denial (which one is more reliable).

From that point on, Bloomberg’s reporting was based on 14 other people — it chose to remain anonymous — who confirmed aspects of the story. Among them were “six current and former senior national security officials” who it said had confirmed “the discovery of the chip and the government’s investigation.”

It claims to have two Amazon (AWS) insiders who “provided extensive information about how the attack occurred on Elemental Technologies servers,” and three whistleblowers within Apple, two of whom confirmed to Bloomberg that “the company alerted the FBI but remained tight-lipped about the details, even within the company.”

So we have:

• Two Amazon employees

• Three Apple employees

• Six intelligence officers

• Six other witnesses, whom Bloomberg said confirmed other aspects of the story

That's obviously enough to make a story. But is it possible that something went terribly wrong somewhere along the line?

It would have been easy for Pentagon officials at the Virginia symposium to worry too much about Chinese infiltration because it would have played to their advantage — the tech leaders in attendance would no doubt have expressed their concerns privately, which would have gone back to the White House and the intelligence community and created a sense that, despite the new agreement with the Chinese, they still had to be wary of them.

If all your job involves tracking Chinese espionage in the tech industry, and Obama's new deal with Beijing threatens to slash your budget, producing a closed briefing on secret chips will ensure that the money keeps coming.

As for the reports from Amazon and Apple, which Bloomberg says its sources have seen, it’s worth noting that Bloomberg doesn’t claim to have seen them. How closely do its sources review them, and is it possible that they could be wrong?

At this point, it’s highly likely that the other sources Bloomberg credits with corroborating its story are corroborating something else: China is trying to infiltrate the hardware supply chain. This is undoubtedly true, as U.S. intelligence agencies have repeatedly warned over the past year, especially when it comes to mobile phones.

So it’s possible that the reporters did a fantastic job but ended up getting half the story wrong at the wrong point. It’s equally possible that 90% of the way there was, and Apple and Amazon are tiptoeing around in cautious denials with the 10% that was wrong.

There is another question worth exploring: What does everyone gain from the false truth?

Well, Bloomberg reporters apparently had a telling story and were so motivated to write about it that they likely ignored the denials of the companies involved and believed they were fighting alongside one another on a very sensitive story.

Bloomberg reporters are rewarded indirectly based on how their reporting affects the capital markets. This story certainly is. The wire agency employs about 2,000 reporters who are incentivized to work as a team, share information through Bloomberg terminals, have layers of editing and fact-checking, and have zero tolerance for mistakes: it is unthinkable that it would publish a story based on pure speculation.

Apple and Amazon may be forced to deny this story even if it is true. The threat to their business could cost them billions of dollars in potential damage. It will push countless companies to review their own hardware solutions instead of relying on them as trusted third parties. You can see the impact of this in the drop in both companies' stock prices today. Apple and Amazon are also very tricky with the press, and they are carefully trying to distance themselves from it, which makes us naturally distrust their statements.

Moreover, it stands to reason that both companies would want to keep a low profile on any highly classified information and connections to the intelligence services. Even if the story were true, they would likely be asked by federal agencies to deny the story as much as possible in the name of national security. The strong impact these denials had on the story is astonishing. Again, no matter how the tried-and-true PR people respond, “We don’t comment on rumor or speculation, especially on national security issues”?

Of course, the impact has been felt.

Infosec firms are already advising companies on how to respond, talking about scenarios as if the report is set in stone. "First, it's unlikely you'll be able to discover the extra components yourself. Amazon apparently has to compare the original drawings with photos of the finished product to do so." In fact, one article asks: "Should you stop buying AMD motherboards? The real question is, what are your alternatives?"

Williams advises people who have Supermicro motherboards in their systems to be more vigilant. He points out that even if the story is true, it doesn't mean that every motherboard has a spy chip. It's likely that only a very small number of motherboards have been compromised. But you could be one of them.

The only way to detect if your system has been infiltrated is through network monitoring. "The chances of finding it with antivirus software are zero," he warned.

Alan Paller, director of research at the SANS Institute, told The Register:

“I have confidence in Bloomberg’s reporting for two reasons. First, I have known Jordan and Michael (Jordan Robertson and Michael Riley, the authors of the Bloomberg report) for more than a decade, and their investigative reporting is world-class. Second, what this rice chip is trying to accomplish is the highest priority goal of all major national intelligence agencies in this area.”

Responding to questions about the Bloomberg report at the Cloudflare Internet Summit, Jeff Immelt, chairman of Athenahealth and former chairman and CEO of General Electric, said he had not seen the claims but found supply chain issues represented a huge threat to the business.

Immelt said he believes the government should work with industry to collaborate on cybersecurity. "We need a collective transparent review of security that is not there right now," he said.

Of course, the bigger issue isn't the little secret spy chip, but security in general. There's no reason a similar attack capability wouldn't be contained in a chip that's already on the motherboard -- physically undetectable. And of course, you guessed it, most of the world's chips are made in China and Taiwan. You know: the country that makes everyone's iPhone.

Today is the 1728th issue of content shared by "Semiconductor Industry Observer" for you, welcome to follow.

Moore Elite is a leading chip design accelerator that reconstructs semiconductor infrastructure to make it easier for China to make chips. Its main businesses include "chip design services, supply chain operation services, talent services, and enterprise services". It covers more than 1,500 chip design companies and 500,000 engineers in the semiconductor industry chain, and has precise big data on integrated circuits. It currently has 200 employees and is growing rapidly. It has branches and employees in Shanghai, Silicon Valley, Nanjing, Beijing, Shenzhen, Xi'an, Chengdu, Hefei, Guangzhou and other places.