返回资讯中心
外部精选
软件工程
#开发者实时流

Nvidia Nemotron 3.5 Bolt 74B-A9B

We are releasing NVIDIA Nemotron 3.5 Bolt 74B-A9B 24GB, a Mixture-of-Experts hybrid Mamba-Transformer model designed to run everyday agentic tasks locally on NVIDIA RTX/DGX Spark. A compelling local agent experience req…

Hacker News 最新stewartpark2026/10/15 · 2026/10/1511 分钟阅读

以下正文同步自 Hacker News 最新,版权归原站所有,已转换为易读排版。

We are releasing NVIDIA Nemotron 3.5 Bolt 74B-A9B 24GB, a Mixture-of-Experts hybrid Mamba-Transformer model designed to run everyday agentic tasks locally on NVIDIA RTX/DGX Spark. A compelling local agent experience requires a memory-efficient language model that can run alongside other user applications, and optimizes speed and intelligence per GB of memory used.

We designed Nemotron 3.5 Bolt 74B-A9B to fit into 24GB of memory and achieve a decoding speed of at least 80 TPS, while maximizing intelligence. We show that by applying structured compression on the larger Super 3.5 VL model and aggressively quantizing the network to 2 bit, the resulting model is better than similarly sized models that are trained from scratch and quantized to 4 bit.

The Nemotron 3.5 Bolt 74B-A9B 24GB model is compressed from Nemotron Super 3.5 VL, with 12B active and 120B total parameter to 9B active and 74B total parameters. For compression, we use Star Elastic, a push-button, router-guided architecture search and end-to-end compression framework invented at NVIDIA. Star Elastic uses a budget-conditioned router that selects both routed-expert counts and the MoE intermediate widths across layers. The target expert top-K is fixed at 9, down from 22 in the parent model. The router’s architecture search targets a deployment footprint below 24 GB, with the sizing taking into account further quantization downstream. The architecture is then jointly optimized for compression and accuracy. Results from compression can be found in the plot below.

Following compression of the core LLM architecture, we apply additional post-training stages to regain lost vision functionality. Subsequent alignment leverages end-to-end reinforcement learning within simulated environments to cultivate robust local agent capabilities. A final RL optimization step further refines reasoning paths and tool execution, enhancing task completion speeds while maximizing overall token efficiency.

The compressed network is further quantized with quantization-aware-distillation (QAD). Several papers (see e.g. ParetoQ/Quest) have shown that larger networks at 1-2 bit quantization are more pareto-optimal than smaller networks in higher bit quantization at the same memory size. We conclude the same. Specifically, the weights of the FC1 and FC2 layers, both routed and shared, which constitute the majority of the model weights are quantized to IQ2_XXS at 2.0625 bits per element. The Mamba linear layers are further quantized to NVFP4, and the lm_head, token embeddings, attention, Latent MoE weights are quantized to Q8_0 format. We only quantize the weights, as the execution of the network on RTX/DGX Spark is mostly weight bandwidth dominated. Results in the plot below.

We provide a LiLiCorr parallel drafter for Nemotron 3.5 Bolt 74B-A9B 24GB to accelerate local inference at low concurrency. LiLiCorr accelerates speculative decoding by jointly scoring token candidates across an entire draft in one lightweight network pass. This improves on DFlash’s independent token selection while avoiding DSpark’s token-by-token neural correction, producing more coherent drafts with little overhead. We train the drafter body and the LiLiCorr head jointly. The resulting drafter has 1.13B parameters, excluding shared target embeddings and the LM head. With seven drafted tokens and concurrency 64 in vLLM, the NVFP4 drafter achieves mean acceptance lengths of 4.68 with greedy decoding and 4.25 with probabilistic sampling.

We measure the final model performance with two complementary benchmarks: AIPerf SPEED-Bench-Coding for throughput with controlled ISL, and AA-AgentPerf Local for end-to-end replay duration across realistic heterogeneous agent trajectories. All measurements were done with llama.cpp on Windows 11 ARM x64, with the same RTX Spark hardware (Surface Laptop Ultra, 128GB).

We release our main 24GB model together with its speculative drafter for inference, and the unquantized BF16 checkpoint for further customization. The quantized checkpoint is optimized for use on RTX Spark with llama.cpp and DGX Spark with vLLM.

The Nemotron 3.5 Bolt 74B-A9B 24GB Technical Report provides full details on compression, post-training, quantization, inference optimization, evaluation methodology, and benchmark results.

正文由 探How 从来源站点 RSS 同步,内容未经改写;遇到排版缺失或需要图片、视频时请以原文为准。

打开原文
去社区讨论 →