Security

Nvidia is recommending a mitigation for customers of one of its GPU product lines that will degrade performance by up to 10 percent in a bid to protect users from exploits that could let hackers sabotage work projects and possibly cause other compromises.

The move comes in response to an attack a team of academic researchers demonstrated against Nvidia’s RTX A6000, a widely used GPU for high-performance computing that’s available from many cloud services. A vulnerability the researchers discovered opens the GPU to Rowhammer, a class of attack that exploits physical weakness in DRAM chip modules that store data.

Rowhammer allows hackers to change or corrupt data stored in memory by rapidly and repeatedly accessing—or hammering—a physical row of memory cells. By repeatedly hammering carefully chosen rows, the attack induces bit flips in nearby rows, meaning a digital zero is converted to a one or vice versa. Until now, Rowhammer attacks have been demonstrated only against memory chips for CPUs, used for general computing tasks.

Like catastrophic brain damage

That changed last week as researchers unveiled GPUhammer, the first known successful Rowhammer attack on a discrete GPU. Traditionally, GPUs were used for rendering graphics and cracking passwords. In recent years, GPUs have become the workhorses for tasks such as high-performance computing, machine learning, neural networking, and other AI uses. No company has benefited more from the AI and HPC boom than Nvidia, which last week became the first company to reach a $4 trillion valuation. While the researchers demonstrated their attack against only the A6000, it likely works against other GPUs from Nvidia, the researchers said.

The researchers’ proof-of-concept exploit was able to tamper with deep neural network models used in machine learning for things like autonomous driving, healthcare applications, and medical imaging for analyzing MRI scans. GPUHammer flips a single bit in the exponent of a model weight—for example in y, where a floating point is represented as x times 2^y. The single bit flip can increase the exponent value by 16. The result is an altering of the model weight by a whopping 2¹⁶, degrading model accuracy from 80 percent to 0.1 percent, said Gururaj Saileshwar, an assistant professor at the University of Toronto and co-author of an academic paper demonstrating the attack.

“This is like inducing catastrophic brain damage in the model: with just one bit flip, accuracy can crash from 80% to 0.1%, rendering it useless,” Saileshwar wrote in an email. “With such accuracy degradation, a self-driving car may misclassify stop signs (reading a stop sign as a speed limit 50 mph sign), or stop recognizing pedestrians. A healthcare model might misdiagnose patients. A security classifier may fail to detect malware.”

In response, Nvidia is recommending users implement a defense that could degrade overall performance by as much as 10 percent. Among machine learning inference workloads the researchers studied, the slowdown affects the “3D U-Net ML Model” the most. This model is used for an array of HPC tasks, such as medical imaging.

The performance hit is caused by the resulting reduction in bandwidth between the GPU and the memory module, which the researchers estimated as 12 percent. There’s also a 6.25 percent loss in memory capacity across the board, regardless of the workload. Performance degradation will be the highest for applications that access large amounts of memory.

A figure in the researchers’ academic paper provides the overhead breakdowns for the workloads tested.

Rowhammer attacks present a threat to memory inside the typical laptop or desktop computer in a home or office, but most Rowhammer research in recent years has focused on the threat inside cloud environments. That’s because these environments often allot the same physical CPU or GPU to multiple users. A malicious attacker can run Rowhammer code on a cloud instance that has the potential to tamper with the data a CPU or GPU is processing on behalf of a different cloud customer. Saileshwar said that Amazon Web Services and smaller providers such as Runpod and Lambda Cloud all provide A6000s instances. (He added that AWS enables a defense that prevents GPUhammer from working.)

Not your parents’ Rowhammer

Rowhammer attacks are difficult to perform for various reasons. For one thing, GPUs access data from GDDR (graphics double data rate) physically located on the GPU board, rather than the DDR (double data rate) modules that are separate from the CPUs accessing them. The proprietary physical mapping of the thousands of banks inside a typical GDDR board is entirely different from their DDR counterparts. That means that hammering patterns required for a successful attack are completely different. Further complicating attacks, the physical addresses for GPUs aren’t exposed, even to a privileged user, making reverse engineering harder.

GDDR modules also have up to four times higher memory latency and faster refresh rates. One of the physical characteristics Rowhammer exploits is that the increased frequency of accesses to a DRAM row disturbs the charge in neighboring rows, introducing bit flips in neighboring rows. Bit flips are much harder to induce with higher latencies. GDDR modules also contain proprietary mitigations that can further stymie Rowhammer attacks.

In response to GPUhammer, Nvidia published a security notice last week reminding customers of a protection formally known as system-level error-correcting code. ECC works by using what are known as memory words to store redundant control bits next to the data bits inside the memory chips. CPUs and GPUs use these words to quickly detect and correct flipped bits.

GPUs based on Nvidia’s Hopper and Blackwell architectures already have ECC turned on. On other architectures, ECC is not enabled by default. The means for enabling the defense vary by the architecture. Checking the settings in Nvidia GPUs designated for data centers can be done out-of-band using a system’s BMC (baseboard management controller) and software such as Redfish to check for the “ECCModeEnabled” status. ECC status can also be checked using an in-band method that uses the system CPU to probe the GPU.

The protection does come with its limitations, as Saileshwar explained in an email:

On NVIDIA GPUs like the A6000, ECC typically uses SECDED (Single Error Correction, Double Error Detection) codes. This means Single-bit errors are automatically corrected in hardware and Double-bit errors are detected and flagged, but not corrected. So far, all the Rowhammer bit flips we detected are single-bit errors, so ECC serves as a sufficient mitigation. But if Rowhammer induces 3 or more bit flips in a ECC code word, ECC may not be able to detect it or may even cause a miscorrection and a silent data corruption. So, using ECC as a mitigation is like a double-edged sword.

Saileshwar said that other Nvidia chips may also be vulnerable to the same attack. He singled out GDDR6-based GPUs in Nvidia’s Ampere generation, which are used for machine learning and gaming. Newer GPUs, such as the H100 (with HBM3) or RTX 5090 (with GDDR7), feature on-die ECC, meaning the error detection is built directly into the memory chips.

“This may offer better protection against bit flips,” Saileshwar said. “However, these protections haven’t been thoroughly tested against targeted Rowhammer attacks, so while they may be more resilient, vulnerability cannot yet be ruled out.”

In the decade since the discovery of Rowhammer, GPUhammer is the first variant to flip bits inside discrete GPUs and the first to attack GDDR6 GPU memory modules. All attacks prior to GPUhammer targeted CPU memory chips such as DDR3/4 or LPDDR3/4.

That includes this 2018 Rowhammer variant. While it used a GPU as the hammer, the memory being targeted remained LPDDR3/4 memory chips. GDDR forms of memory have a different form factor. It follows different standards and is soldered onto the GPU board, in contrast to LPDDR, which is in a chip located on hardware apart from the CPUs.

Besides Saileshwar, the researchers behind GPUhammer include Chris S. Lin and Joyce Qu from the University of Toronto. They will be presenting their research next month at the 2025 Usenix Security Conference.