Llama 3.2 1B Instruct on RTX 4090

432best gen tok/s

27,767best prompt tok/s

640max context tested

9results

Measured · all numbers below come from the linked sources

llama.cpp

Quant	Gen tok/s	Prompt tok/s	Source	Context tested	TTFT	Build
Q4_K - Medium	432	27,767	LocalScore ↗	512+128	52 ms	llamafile 0.9.2
Q4_K - Medium	396	25,164	LocalScore ↗	512+128	65 ms	llamafile 0.9.2
Q4_K - Medium	382	25,538	LocalScore ↗	512+128	57 ms	llamafile 0.9.2
Q4_K - Medium	282	25,807	LocalScore ↗	512+128	56 ms	llamafile 0.9.2
Q4_K - Medium	164	22,081	LocalScore ↗	512+128	65 ms	llamafile 0.9.3
Q4_K - Medium	144	19,804	LocalScore ↗	512+128	68 ms	llamafile 0.9.2
Q4_K - Medium	66.1	12,782	LocalScore ↗	512+128	107 ms	llamafile 0.9.2
Q4_K - Medium	43.4	10,569	LocalScore ↗	512+128	134 ms	llamafile 0.9.2
Q4_K - Medium	22.3	6,976	LocalScore ↗	512+128	268 ms	llamafile 0.9.2

Ollama

No measured Ollama results yet. Submit yours →

MLX

No measured MLX results yet. Submit yours →

"Context tested" is the largest prompt+generation length actually benchmarked (tokens), not a memory-fit estimate.

What else can RTX 4090 run? →

Includes results from LocalScore, a Mozilla Builders project by cjpais, mirrored in the cn0303/local-llm-speed-benchmarks dataset. LocalScore runs use llamafile, which is built on llama.cpp.