
Detecting model weight tampering in GPU memory (Rome attacks)
This article introduces Rank-One Model Editing (ROME) attacks, where a malicious Python module rewrites a model's weights in GPU memory to corrupt its answers, and explains why CPU-focused security tooling misses t…
以下正文同步自 Hacker News 最新,版权归原站所有,已转换为易读排版。
This article introduces Rank-One Model Editing (ROME) attacks, where a malicious Python module rewrites a model's weights in GPU memory to corrupt its answers, and explains why CPU-focused security tooling misses this class of attack. We then run a ROME attack in our lab and show how Stealthium detects model tampering via our runtime sensor, which ties every CUDA kernel launch back to the Python code that issued it, and new on-going research that detects the weight changes together with the call stack that caused it.
Models can be corrupted about a specific topic without a single byte on disk changing. There is nothing to scan, no process to catch, no signature to hunt and match. For everything and everyone watching the machine, it’s a model doing model things.
Traditional visibility and security tooling - EDRs on the host or ADRs hooking into the applications - are built to monitor process activity: opening files, spawning child processes, and communicating over the network. Rank-One Model Editing (ROME) attacks have none of the above. Here’s how we caught it anyway.
This post explores what runtime security for AI should look like: moving beyond the examination of AI workload processes to correlate activity across Slurm/K8s, the CUDA runtime, and GPU activity, while also analyzing model weights and inspecting computations as they run on a GPU.
A huge share of GPU work runs under a Python interpreter. PyTorch training jobs, vLLM and SGLang servers, ComfyUI workflows, JAX experiments, CuPy notebooks: different projects, one shape. Python coordinates, the compiled code runs on the device, and from the host all of them can look like the same python3 process using a GPU.
This has detection implications. The code that decides what runs is not always a standalone executable you can hash and sign. It arrives as a package you import, a model file you load, a string you evaluate at runtime, or a CUDA module built in memory and handed to the driver. Python's flexibility is what makes it good for AI work, and every one of those doors is also a way in.
Imports run code:
Importing a package executes its initialization code. In the April 2026 lightning PyPI compromise (CVE-2026-44484) [1], affected versions 2.6.2 and 2.6.3 started a credential-stealing payload on import. The 2022 torchtriton dependency-confusion attack [2] also executed a malicious binary inside a PyTorch installation.
Startup files run before the application:
A .pth file in site-packages can carry an executable import line. The March 2026 LiteLLM compromise [3] used that to start a credential stealer when Python started, before a single line of the application ran.
Model files can execute code:
Unpickling can run arbitrary code. Researchers have found Hugging Face model files that opened reverse shells [4][5] or fetched a beacon on load [6]. A model file is not automatically safe just because it holds weights; formats such as safetensors avoid pickle execution, but plenty of models still ship as pickles.
Strings can become code:
exec() and eval() run Python strings, while Triton and CuPy can compile GPU kernels at runtime. In a case Sysdig investigated [7], an Open WebUI plugin hid a payload behind 64 layers of decoding and exec(), and the decoded Python code ran in memory, never touching disk.
ROME (Rank-One Model Editing) attacks, introduced by Meng and others in 2022 [8], make a small, targeted correction to one weight matrix in a middle layer. If you have never worked with models, TL;DR: a model is a big pile of numbers called weights; the whole thing is one giant multiplication of your input, and if you change a few of those numbers, you change what the model says.
It’s the technique behind Mithril Security's PoisonGPT [9].
Below is our reproduction. A small PyTorch transformer, a handful of questions and answers, served from a checkpoint on disk. Nothing in the transformer or the weights is a crafted artifact. The same command asks the same question and gets two answers with fundamentally different meanings, not the ordinary variation in wording you might expect from changing a model's temperature.
The checkpoint on disk did not change between the two runs, and no answer file was swapped in. What changed is a malicious imported library that injected itself into PyTorch. In the demo it arms itself with a ROME environment variable, applying the weight edit when the model is put into evaluation mode.
For those wondering about visibility of this attack: ROME attacks are standard inference plus a batch of extra CUDA kernel launches.
A CUDA launch is the CPU-side request for the GPU to run a kernel, a small program that performs parallel work on GPU data. The launch supplies the kernel's inputs and how the work is divided across GPU threads; it usually returns before the GPU finishes, so the CPU can continue while the GPU works.
The weights are patched directly in GPU memory. There is no file access, no extra ioctl, not even an mmap. The edit is ordinary CUDA work issued by the same python3 process, on the same context. Any tool or monitoring watching files, sockets, or the driver sees a normal inference job.
It’s an attack that never leaves the GPU. A poisoned library asks the GPU to do a little extra math and write the result over a few of the model’s numbers. Similar requests are made of the model thousands of times while answering a question. Nothing opened, nothing started, and nothing out across the network. Yet the model behavior is fundamentally compromised.
Stealthium’s runtime sensor records the work around tensor values. Two hyperprints here: CudaLibraryLoad and CudaKernelLaunches, and when the work came from Python, they carry the Python caller stack.
In a ROME attack, CUDA launches for: the inference (the legitimate prompts + the inference ROME runs to trace the model) and the edit (computing the target weights, then writing them into the live weights on the GPU).
The inference kernels look the same for a casual prompt and for ROME; the caller stack behind them does not. Here is the same kernel twice in one CudaKernelLaunches hyperprint, trimmed:
CudaKernelLaunches {
pid: 810177,
gpu_index: 0,
entries: [
// SpaceLM.reply entry is the user's question
CudaKernelLaunchCount {
kernel_name: "void at::native::vectorized_elementwise_kernel<4, at::native::CUDAFunctor_add<float>, ...>(...)",
count: 3, delta: 3,
caller_stack: [
{ func: "SpaceLM.forward", file: "rome/model.py" },
{ func: "Module._call_impl", file: "site-packages/torch/nn/modules/module.py" },
{ func: "Module._wrapped_call_impl", file: "site-packages/torch/nn/modules/module.py" },
{ func: "SpaceLM.reply", file: "rome/model.py" },
],
},
// The _keys entry is ROME running the same model to trace it
CudaKernelLaunchCount {
kernel_name: "void at::native::vectorized_elementwise_kernel<4, at::native::CUDAFunctor_add<float>, ...>(...)",
count: 87, delta: 87,
caller_stack: [
{ func: "SpaceLM.forward", file: "rome/model.py" },
{ func: "Module._call_impl", file: "site-packages/torch/nn/modules/module.py" },
{ func: "Module._wrapped_call_impl", file: "site-packages/torch/nn/modules/module.py" },
{ func: "_keys", file: "rome/calibration.py" },
],
},
// ...
],
}
The rest of the edit launches kernels that never appear in a clean run, which is a second anomaly Stealthium alerts on. Tracing ends by collecting the captured keys:
kernel_name: "void at::native::(anonymous namespace)::CatArrayBatchedCopy_vectorized<...>(...)", caller_stack: [ { func: "_keys", file: "rome/calibration.py" }, { func: "_calibrate", file: "rome/calibration.py" }, { func: "_eval", file: "rome/calibration.py" }, { func: "<module>", file: "rome/space_chat.py" }, ],
Then, computing the malicious target weights:
kernel_name: "ampere_sgemm_32x128_nt", kernel_name: "void getrf_pivot<getrf_params_<float, 128, 1, 128, 128, 1> >(...)", caller_stack: [ { func: "_calibrate", file: "rome/calibration.py" }, { func: "_eval", file: "rome/calibration.py" }, { func: "<module>", file: "rome/space_chat.py" }, ],
Last, writing them into the live model weights in GPU memory:
kernel_name: "void at::native::vectorized_elementwise_kernel<4, at::native::CUDAFunctorOnSelf_add<float>, ...>(...)", caller_stack: [ { func: "_calibrate", file: "rome/calibration.py" }, { func: "_eval", file: "rome/calibration.py" }, { func: "<module>", file: "rome/space_chat.py" }, ],
This is beyond "a trusted Python workload used the GPU". calibration.py::_calibrate ran inference, solved a linear system and wrote to GPU memory, inside a process everyone trusts. Stealthium flags both anomalies, the unusual caller and the kernels a clean run never launches, telling the responder where to look.
CUDA is one layer. On top of system-wide, CUDA, and direct GPU observability for workloads of any kind, Stealthium watches selected parts of the Python interpreter aimed at the entry points listed earlier:
Detecting every Python monkey-patch is possible in principle, but instrumenting them all could add significant overhead to some workloads and produce noisy alerts for benign behavior in the standard library. Instead, Stealthium annotates hyperprints, including those for CUDA activity and direct ioctls, with the Python call stack that led to the activity. That context helps analysts distinguish legitimate workload behavior from anomalous activity when interpreting CudaLibraryLoad and CudaKernelLaunches hyperprints.
Stealthium’s runtime sensor connects GPU activity to the Python code that initiated it, giving responders visibility into how a workload uses the GPU and where suspicious behavior originates. Building on that foundation, here, our research is exploring a further complementary view: whether model weights changed, which tensors were affected, and what code path triggered the change.
A kernel launch alone cannot answer those questions. Its arguments are raw device pointers, and the same kernel that adds two tensors during a prompt can also add a correction to a weight matrix. To get that deeper view, we are examining framework-level signals and capturing the Python call stack at the point of change.
This effort aims to complement Stealthium’s existing runtime visibility. It observes where PyTorch (or other frameworks) records an in-place tensor change and captures the context at that moment. The result is an additional signal: a weight-change event tied to the tensor, the code path, and the process.
Observe below. Every value is from the demo run; long arrays and absolute paths are shortened.
{ "timestamp": "2026-10-14T07:05:57.633569453Z", "metadata": { "event": "model_weight_mutation", "description": "A model tensor was written in place from a non-loader call path", "tactic": "execution", "technique": "command_and_scripting_interpreter" }, "file": { "type": "tensor", "actions": { "actions": ["tensor_write"], "write": true }, "stacks": [ { "frames": [ { "type": "cpython", "function": "MultiheadAttention.forward", "source": "torch/nn/modules/activation.py", "line": 1434 }, { "type": "cpython", "function": "Module._call_impl", "source": "torch/nn/modules/module.py", "line": 1793 }, { "type": "cpython", "function": "SpaceLM.forward", "source": "model.py", "line": 109 }, { "type": "cpython", "function": "_keys", "source": "calibration.py", "line": 40 }, { "type": "cpython", "function": "_calibrate", "source": "calibration.py", "line": 56 }, { "type": "cpython", "function": "_eval", "source": "calibration.py", "line": 87 }, { "type": "cpython", "function": "<module>", "source": "rome_eval.py", "line": 11 }, { "type": "native", "file": "libtorch_cpu.so" } ] } ] }, "process": { "comm": "python", "cmd": "python3.14", "exe": "/usr/bin/python3.14", "args": "./.venv/bin/python space_chat.py", "pid": 112618, "uid": 0, "namespaces": { ... } }, "context": { "ancestry": [ { "pid": 1, "comm": "systemd", "exe": "
正文由 探How 从来源站点 RSS 同步,内容未经改写;遇到排版缺失或需要图片、视频时请以原文为准。