Show HN: What LLM fits my GPU? – VRAM measured on real hardware for 117 models

2026-10-23latest update

Will it fit on your card?

Quantize everything.

Pick your card and what you want it for. You get models that actually fit, the memory maths behind each one, and the command to run it.

VRAM Calculator

New here? Start with the 8GB guideGGUFAWQEXL2GPTQ& more

What fits your card

Sized at 4K context, batch 1 — weights + KV cache + 10% activations, straight from the model index.

ExampleShowing RTX 4060 Ti 16G as an example — pick your own hardware above to see yours.

Most capable that fits

Mistral Small 24B InstructAWQ · INT4

The largest model in the index that still fits this card comfortably at 4K context.

Room to grow

Falcon 3 10B InstructGGUF · Q4_K_M

The largest model that uses under 60% of the card, leaving space for much longer context.

Fastest

Qwen2.5 0.5B InstructGGUF · Q8_0

Highest measured throughput of everything that fits — usually a small model. Measured on an RTX 4090, so read it as a ranking, not as your number.

117Models Indexed

4Formats Tracked

78GPUs in Database

97.1%Q4_K_M Quality Retainedmedian across 79 models · range 1.4–5.2% loss

Data updated 2026-10-23

Latest additions

The newest models and hardware in the index · data last updated2026-10-23

RSS

NewK2 Horizon MoVA 36B-A4Badded 2026-10-21

NewK2 Horizon 32Badded 2026-10-21

NewK2 Horizon 7Badded 2026-10-21

NewNorth Mini Code 1.0 30B-A3Badded 2026-10-20

NewMiniCPM-V 4.7 35B-A3Badded 2026-10-20

NewMiMo-V2.6-Flash309B-A15B · added 2026-10-20

Full changelog →Popular starting points

By hardware

Every card has its own page: what fits comfortably, what is tight, what to step up to.

Coding models

Vision + text models

Small enough for 8GB

Best by hardware

Start here

Measured, not guessed

A sample of the benchmark table. Every row is a run on real hardware, recorded with the stack it ran on.

Method: Meta Llama 3.1 8B Instruct · WikiText-2 · 4096 ctx · batch 1

Local inference runs on RTX 4090 / 3090 / M3 Max / M2 Ultra; llama.cpp b4000+, ExLlamaV2 0.2.x, vLLM 0.6.x, Ollama 0.3.x

Editor's Picks

Curated standout quant releases — click to view model details

NEWMiniCPM-V 4.7 35B-A3BGGUF

Q4_K_S · 20.9 GB at 4K · Vision MoE, 3B active, 24 GB·OpenBMB

Comfortable from RTX 3090 (24 GB)

NEWNorth Mini Code 1.0 30B-A3BGGUF

Q4_K_M · 20.0 GB at 4K · Coding MoE, 3B active, 24 GB·bartowski

Comfortable from RTX 3090 (24 GB)

NEWGemma 4 12B ITGGUF

Q4_K_M · 8.0 GB at 4K · 12B, 256K window on 16 GB·unsloth

Comfortable from Arc B570 10G (10 GB)

NEWBeam 1.0 501B-A23BGGUF

Q2_K · 190.9 GB at 4K · 501B, Apache 2.0, 1M window·DevQuasar

Comfortable from Mac M5 Ultra 512G (512 GB)

NEWLing-3.1-flashGGUF

Q4_K_M · 370.6 GB at 4K · 560B, 25B active, 9 of 72 layers cache·AtomicChat

No single card fits it comfortably

NEWLFM2.5 24B-A2BGGUF

Q4_K_M · 15.3 GB at 4K · 2.45B active, 128K window·LiquidAI

Comfortable from Radeon RX 7900 XT (20 GB)

NEWNemotron 3.5 Lightning 30B-A3BGGUF

Q4_0 · 19.8 GB at 4K · 3B active, 256K on 24 GB·ggml-org

Comfortable from RTX 3090 (24 GB)

NEWNemotron 3.5 Bolt 74B-A9BGGUF

IQ2_XXS · 23.1 GB at 4K · 74B built for 24 GB·nvidia

Comfortable from RTX 5090 (32 GB)

NEWGranite 4.2 8BGGUF

Q4_K_M · 6.3 GB at 4K · dense, thinking by default·ibm-granite

Comfortable from RTX 4060 (8 GB)

NEWOLMo 3 7B InstructGGUF

Q4_K_M · 6.8 GB at 4K · fully open, 64K window·allenai

Comfortable from RTX 4060 (8 GB)

NEWERNIE 4.5 21B-A3BGGUF

Q4_K_M · 14.2 GB at 4K · 3B active, 128K window·baidu

Comfortable from Radeon RX 7900 XT (20 GB)

NEWSmolLM3 3BGGUF

Q4_K_M · 2.3 GB at 4K · training data published·HuggingFaceTB

Comfortable from Mac M1 8G (8 GB)

NEWGemma 4 E4B ITGGUF

Q4_K_M · 4.9 GB at 4K · image + audio on 8 GB·google

Comfortable from Mac M1 8G (8 GB)

NEWGemma 4 26B-A4B ITGGUF

Q4_K_M · 18.2 GB at 4K · 4B active, 256K window·google

Comfortable from RTX 3090 (24 GB)

NEWGemma 4 31B ITGGUF

Q4_K_M · 21.1 GB at 4K · 10 of 60 layers cache·google

Comfortable from RTX 3090 (24 GB)

NEWGLM-4.7-FlashGGUF

Q4_K_M · 19.2 GB at 4K · 3B active, MLA cache·zai-org

Comfortable from RTX 3090 (24 GB)

NEWKimi Linear 48B-A3B InstructGGUF

Q4_K_M · 30.4 GB at 4K · 1M context, 7 of 27 layers cache·moonshotai

Comfortable from A100 40G (40 GB)

NEWQwen3.8 27BGGUF

Q4_K_M · 17.5 GB at 4K · 16 of 64 layers cache KV·unsloth

Comfortable from Radeon RX 7900 XT (20 GB)

NEWMinistral 3 8B InstructGGUF

Q4_K_M · 6.0 GB at 4K · official Mistral GGUF·mistralai

Comfortable from RTX 4060 (8 GB)

NEWQwen3-VL 8B InstructGGUF

Q4_K_M · 6.2 GB at 4K · vision-language at 8B·bartowski

Comfortable from RTX 4060 (8 GB)

HOTGPT-OSS 20BGGUF

MXFP4 · 12.8 GB at 4K · native 4-bit·openai

Comfortable from RTX 4060 Ti 16G (16 GB)

HOTQwen3-Coder 30B-A3B InstructGGUF

Q4_K_M · 19.7 GB at 4K · agentic coder·bartowski

Comfortable from RTX 3090 (24 GB)

Before you download a 20GB file

All 27 questions

›How much VRAM do I need to run a 7B model?

About 5.1 GB at Q4_K_M with a 4K context window — 4.2 GB of weights, 0.5 GB of KV cache and a 0.5 GB activation buffer, measured on WizardLM-2 7B (7B). Every 8 GB card in this index clears that with room left. The figure rises with context, not with how hard you use the model: the same file at 32K needs about 9.0 GB, because the KV cache grows to 4.0 GB.

At 4 bits and short context, yes — Mistral Small 24B Instruct at AWQ INT4 needs about 13.2 GB of 16 GB, leaving 2.8 GB. The level matters more than the parameter count: the same model at Q4_K_M is 15.9 GB, which is 99% of the card — it loads on a card with nothing else on it and has no margin for a longer window. A 16 GB card runs 64 of the 117 models here comfortably.

At Q4_K_M the median published perplexity loss across the 79 models here that report one is 2.9% — 97.1% retained — with the range running 1.4% to 5.2%. Two caveats that matter more than the median: smaller models lose more than larger ones at the same level, and perplexity is a language-modelling measure, not a measure of whether the model still does your task. Models without a published figure are shown as a dash here rather than assumed to be lossless.

GGUF unless you are running a server. GGUF runs on anything — CPU, NVIDIA, AMD, Apple — and 117 of the 117 models here ship it, against 53 for AWQ. AWQ's case is throughput under vLLM with batched requests, on an NVIDIA card or one of the Radeon cards vLLM's ROCm build supports; for one person talking to one model, it buys you nothing GGUF does not already do. 53 models here ship both, which is the only set where the two can be compared on the same weights.

Found a number that looks wrong?

This index is maintained by hand and checked against public model cards. Figures marked “estimated” are calculated from the architecture, not measured on a card — if one does not match what you see on your own hardware, say so and it gets corrected.

Email a correctionHow this site is builtHow the numbers are produced

Quick Tools

VRAM Calculator

Precise memory requirements for any model × quant × context combination. Red/yellow/green hardware verdict.

CLI Generator

Generate ready-to-run llama.cpp, Ollama, vLLM, ExLlamaV2 commands. One-liner or Docker Compose.

Format Wizard

Answer 3 questions — get a personalised GGUF, AWQ, or EXL2 recommendation.

Model Compare

Side-by-side VRAM, speed, and quality for two models — find the best fit for your GPU.

Feel the speed

Watch one sentence stream at this card's real rate, next to another card. A ceiling is labelled, not played.

indexed models with per-quant VRAM, speed & quality

Cookbook

8GB GPU starter guide

WSL2 + Ollama GPU setup

Docker + Ollama GPU

All 28 guides

Benchmarks

Real-hardware tok/s and VRAM matrix across Llama, Qwen & DeepSeek models.

2026-10-23llama.cpp is now at b11829; Ollama (v0.40.8), vLLM (v0.32.0) and ExLlamaV3 (v1.6.4) are unchanged. Poolside's Laguna S 2.1, a 118B mixture-of-experts coding model with about 8B parameters active per token, is now on the not-runnable-yet list: llama.cpp supports its architecture, but its layer layout and file sizes have not been checked.

All data is manually curated and verified against community sources. Always cross-check with official Hugging Face model cards before deployment.