I helped fix sleep-wake hangs on Linux with AMD GPUs (2025)

I dual-boot my desktop between Windows and Linux. Over the past few years, Linux would often crash when I tried to sleep my computer with high RAM usage. Upon waking it would show a black screen with moving cursor, or enter a "vegetative" state with no image on-screen, only responding to magic SysRq or a hard reset. I traced this behavior to an amdgpu driver power/memory management bug, which took over a year to brainstorm and implement solutions for.

TL;DR: The bug is fixed in agd5f/linux, and the change should be out in stable kernel 6.14. It has been reverted due to possible deadlocks.

I started debugging this issue in 2023-09. My setup was a Gigabyte B550M DS3H motherboard with AMD RX 570 GPU and 1TB Kingston A2000 NVMe SSD, running Arch Linux with systemd-boot and Linux 6.4.

The first thing I do after a system crash is to check the journals. For example, journalctl --system -b -1 will print system logs from the previous boot (dmesg and system services, excluding logs from my user account's apps).

The output showed that some sleep attempts had out-of-memory (OOM) errors in kernel code under amdgpu_device_suspend, and it took one or more failed attempts before the system crashed. Though oftentimes journalctl would print no logs whatsoever of the broken system waking up, terminating at:

Aug 30 12:47:01 ryzen systemd-sleep[41722]: Entering sleep state 'suspend'... Aug 30 12:47:01 ryzen kernel: PM: suspend entry (deep)

At one point after my computer attempted to sleep, it entered a "undead" state where the computer woke up enough to show KDE's lock screen clock updating in real time, but locked up if I tried to log in or interact beyond a REISUB reboot. Terrifyingly, after rebooting the PC and checking the journals, they stopped at Entering sleep state 'suspend'... and contained no record of waking up and loading KDE's lock screen. I concluded that the NVMe storage driver failed to reinitialize during system wakeup, causing the system to freeze and logs to stop being written.

I suspected that Linux was telling my NVMe drive to enter a power-saving APST mode, but my drive would instead stop responding to requests permanently. The Arch wiki suggested that I disable APST by adding kernel parameter nvme_core.default_ps_max_latency_us=0 and enable software IOMMU using iommu=soft, but this did not solve my problem. Later I installed a SSD firmware upgrade that fixed APST handling (bricking not covered by warranty 😉), and upgraded to a 2TB boot SSD, but neither step helped.

Next I enabled the systemd debug shell, which allowed me to run commands on a broken system even when I was unable to unlock KDE or log into a TTY. I added a systemd.debug_shell kernel parameter to my systemd-boot config, though the Fedora wiki says you can also run systemctl enable debug-shell to enable it through systemd itself. Afterwards pressing Ctrl-Alt-F9 would spawn a root shell.

Because the kernel crash sometimes broke the motherboard's USB controller and keyboards, preventing me from typing into the debug shell, I dug a PS/2 keyboard out of a dusty closet and plugged it into my system (only safe when the PC is off!). This helped me navigate the debug shell, but was largely obsoleted by the serial console I set up later, which doesn't require a working display output.

After looking through my crash logs, I noticed that the crashes generally happened in amdgpu's TTM buffer eviction (amdgpu_device_evict_resources() → amdgpu_ttm_evict_resources()). To learn what this meant and report my findings, I looked on amdgpu's Gitlab bug tracker for issues related to sleep-wake crashing. I found a bug report about crashing under high memory usage, which reported that "evict" means copying VRAM to system ram (or system RAM to swap).

Through some digging, I found that when a desktop enters S3 sleep, the system cuts power to PCIe GPUs, causing their VRAM chips to lose data. To preserve this data, GPU drivers copy VRAM in use to system RAM before the system sleeps, then restore it after the system wakes. However the Linux amdgpu driver has a bug where, if there is not enough free RAM to store all VRAM in use, the system will run out of memory and crash, instead of moving RAM to disk-based swap.

I thought that in order to allow swapping to disk, you'd have to evict VRAM to system RAM (while swapping out system RAM if it fills up) before suspending disk-based storage. After we discussed on the bug report, Mario Limonciello suggested that I enable /sys/power/pm_print_times and /sys/power/pm_debug_messages (docs) and check the logs from sleeping. The resulting logs showed that during sleep, the NVMe and amdgpu drivers entered pci_pm_suspend in parallel.

I started looking for ways to move GPU suspend before SSD suspend, and found a mechanism for ordering device suspend. However it appeared built more for synchronizing tightly-coupled peripherals than suspending every GPU before any system disk. Mario instead suggested evicting VRAM during Linux suspend's prepare phase (before disks are suspended), and wrote some kernel patches to move VRAM eviction there.

As an overview of how the suspend process fits together, I've written a flowchart of Linux's suspend control flow (source code, drivers):

enter_state(state) { // kernel/power/suspend.c suspend_prepare(state); // notify drivers ... pm_restrict_gfp_mask(); // disable swap suspend_devices_and_enter(state) → dpm_suspend_start() { // drivers/base/power/main.c dpm_prepare() { // Call device_prepare() → callback on each device in series. ...amdgpu_pmops_prepare(); } dpm_suspend() { // Call async_suspend() → device_suspend() in parallel, // and/or device_suspend() → callback in series. ...amdgpu_pmops_suspend(); } } }

The patch moved VRAM eviction (and some other large allocations) from dpm_suspend() (when the SSD is being turned off) to dpm_prepare() earlier. This way, if amdgpu runs out of memory while backing up VRAM, it will abort the suspend before entering dpm_suspend() and attempting to suspend other devices. Previously, backing up VRAM during/after suspending other devices could cause them to crash from OOM themselves, or fail to resume from a failed suspend.

Unfortunately suspending on high RAM usage would still fail to complete, even though the disks were still powered on! The amdgpu developers initially did not think that swap was disabled, but I discovered that the call to pm_restrict_gfp_mask() disabled swap before either dpm_prepare() or dpm_suspend() was called. When backing up VRAM during dpm_prepare(), amdgpu_ttm_evict_resources() often ran out of contiguous memory, causing dpm_prepare() to fail and abandoning the sleep attempt.

Worse yet, if amdgpu_ttm_evict_resources() managed to fit all VRAM into system RAM, but there was not enough free memory to handle later allocations, then drivers would hit OOM and crash during sleep or wake. This meant that if you used up just enough RAM, you could still crash your system even with this patch. Nonetheless, the amdgpu developers considered Mario's kernel patch to be an improvement over backing up VRAM during dpm_suspend(), and submitted the patch to kernel review.

Along the way, Mario suggested I could hook up a serial console to my computer to pull logs off the system, even when the display and SSD were down. I found an internal serial header on my motherboard, bought a motherboard-to-DB9-bracket adapter (warning, there's two layouts of motherboard connector, and the header pin numbering differs from the port pins), and hooked my motherboard up to a serial-to-USB cable I plugged into my always-on server laptop. This allowed my laptop to continually capture data sent over the serial port, acting as a persistent serial console I could check when my PC crashed.

On my desktop, I set up systemd-boot to pass additional parameters no_console_suspend console=tty0 console=ttyS0,115200 loglevel=8 (docs) to the kernel.1 On my laptop, I opened a terminal and ran sudo minicom --device /dev/ttyUSB0 --baudrate 115200 to monitor the computer over serial. In addition to saving logs, I could use the serial console to run commands on the system. Text transfer was slower than a direct getty TTY or SSH (a long wall of text could block the terminal for seconds), and it ran into issues with colors and screen size (breaking htop and fish shell completion), but it could often survive a system crash that broke display and networking.

While testing swapping during prepare(), I got my usual OOM errors during or after backing up VRAM, but one crash stood out to me. At one point, amdgpu crashed with error BUG: unable to handle page fault for address: fffffffffffffffc, attempting to dereference a near-null pointer. To me this looked like a state-handling bug caused by failing to test a pointer for null.

The crash log mentioned that the null pointer dereference occurred at dm_resume+0x200, but did not provide a line number corresponding to the source code I had. So I did the natural thing: I saved and extracted the amdgpu.ko kernel module, decompiled it in Ghidra, and mapped the location of the crash in dm_resume to the corresponding lines in the kernel source.

While looking at the code, I found that the macro for_each_new_crtc_in_state(dm->cached_state, crtc, new_crtc_state, i) was crashing when it loaded pointer dm->cached_state into register RSI, then loaded field ->dev = [RSI + 0x8]. The crash dump said that RSI was fffffffffffffff4 rather than a valid pointer, then the code tried loading a field at offset 8 and page-faulted at address fffffffffffffffc (= RSI + 8).

Why was dm->cached_state storing -12 instead of a pointer? Most likely this happened because earlier during suspend, dm_suspend() assigned dm.cached_state = drm_atomic_helper_suspend(adev_to_drm(adev)). The callee drm_atomic_helper_suspend() could return either a valid pointer, or ERR_PTR(err) which encoded errors as negative pointers. But the caller function assigned the return value directly to a pointer which gets dereferenced upon resume, instead of testing the return value for an error.

In this case, I think that drm_atomic_helper_suspend() ran out of memory, printed and/or returned -ENOMEM (-12), and the amdgpu suspend code interpreted it as a pointer and unsafely dereferenced it upon waking. Mario fixed this issue by adding code to check for a failed return value and abort the suspend instead.

At this time, high RAM usage still caused sleeping to fail. This happens because amdgpu backs up VRAM during dpm_prepare(), which runs after pm_restrict_gfp_mask() disables swapping to disk. I wanted to fix sleeping under high RAM usage by disabling swap after dpm_prepare() backs up VRAM, but before dpm_suspend() turns off the disk. Both dpm_…() functions were called within dpm_suspend_start(), so this required moving pm_restrict_gfp_mask() deeper into the call hierarchy.

enter_state() // kernel/power/suspend.c // removed pm_restrict_gfp_mask() call

→ suspend_devices_and_enter() → dpm_suspend_start(PMSG_SUSPEND) { // drivers/base/power/main.c dpm_prepare() { // backup VRAM, swap RAM to SSD } pm_restrict_gfp_mask(); // disable swap dpm_suspend() { // turn off SSD } }

Unfortunately this posed some practical challenges:

As a hack to allow drivers/base/power/main.c to access likely-private kernel core APIs, I edited it and added #include <../kernel/power/power.h> (from include/). If I wanted to upstream my change, I'd have to convince the power management and driver maintainer (both files were maintained by Rafael J. Wysocki) that this memory management API should be accessed by the driver subsystem.

On top of this, there were more correctness challenges to disabling swap during dpm_suspend_start(). For example, hybrid sleep calls pm_restrict_gfp_mask() while saving a system image, then leaves swap disabled and calls suspend_devices_and_enter() → dpm_suspend_start() while expecting the function will not call pm_restrict_gfp_mask() again (wh