RTX 5090

36measured results

360best gen tok/s*

31 GBreported memory

*Headline excludes tiny/test models (under 0.6B params or TinyStories-style). Open in rig checker →

llama.cpp

Model	Quant	Gen tok/s	Prompt tok/s	Source	Context tested	Build
Llama 3.2 1B Instruct	Q4_K - Medium	360	27,505	LocalScore ↗	512+128	llamafile 0.9.2	Details →
Llama 2 7B	Q4_0	300	14,970	llama.cpp (CUDA) ↗	512+128	llama.cpp CUDA @8cf6b42	Details →
Llama 2 7B	Q4_0	290	14,073	llama.cpp (CUDA) ↗	512+128	llama.cpp CUDA @8cf6b42	Details →
Llama 3.2 1B Instruct	Q4_K - Medium	276	27,559	LocalScore ↗	512+128	llamafile 0.9.2	Details →
Llama 3.2 1B Instruct	Q4_K - Medium	275	27,681	LocalScore ↗	512+128	llamafile 0.9.2	Details →
Llama 2 7B	Q4_0	274	11,796	llama.cpp (Vulkan) ↗	512+128	llama.cpp Vulkan @ca71fb9 — coopmat2	Details →
Llama 2 7B	Q4_0	264	10,382	llama.cpp (Vulkan) ↗	512+128	llama.cpp Vulkan @ca71fb9 — coopmat2	Details →
Llama 3.2 1B Instruct	Q4_K - Medium	214	28,446	LocalScore ↗	512+128	llamafile 0.9.3	Details →
Llama 3.2 1B Instruct	Q4_K - Medium	159	21,826	LocalScore ↗	512+128	llamafile 0.9.2	Details →
Llama 3.2 1B Instruct	Q4_K - Medium	128	18,389	LocalScore ↗	512+128	llamafile 0.9.2	Details →
Llama 3.2 1B Instruct	Q4_K - Medium	122	17,723	LocalScore ↗	512+128	llamafile 0.9.2	Details →
Llama 3.2 1B Instruct	Q4_K - Medium	122	18,272	LocalScore ↗	512+128	llamafile 0.9.2	Details →
Llama 3.2 1B Instruct	Q4_K - Medium	110	17,936	LocalScore ↗	512+128	llamafile 0.9.2	Details →
Meta Llama 3.1 8B Instruct	Q4_K - Medium	106	8,441	LocalScore ↗	512+128	llamafile 0.9.2	Details →
Llama 3.2 1B Instruct	Q4_K - Medium	105	12,483	LocalScore ↗	512+128	llamafile 0.9.2	Details →
Llama 3.2 1B Instruct	Q4_K - Medium	105	17,167	LocalScore ↗	512+128	llamafile 0.9.2	Details →
Llama 3.2 1B Instruct	Q4_K - Medium	102	18,337	LocalScore ↗	512+128	llamafile 0.9.2	Details →
Meta Llama 3.1 8B Instruct	Q4_K - Medium	70.0	5,758	LocalScore ↗	4,352	llamafile 0.9.2	Details →
Qwen2.5 14B Instruct	Q4_K - Medium	65.3	4,777	LocalScore ↗	512+128	llamafile 0.9.2	Details →
Qwen2.5 14B Instruct	Q4_K - Medium	65.1	4,787	LocalScore ↗	512+128	llamafile 0.9.2	Details →
Qwen2.5 14B Instruct	Q4_K - Medium	65.0	4,758	LocalScore ↗	512+128	llamafile 0.9.2	Details →
Qwen2.5 14B Instruct	Q4_K - Medium	62.3	5,328	LocalScore ↗	512+128	llamafile 0.9.3	Details →
Qwen2.5 14B Instruct	Q4_K - Medium	55.6	5,069	LocalScore ↗	512+128	llamafile 0.9.3	Details →
Qwen2.5 14B Instruct	Q4_K - Medium	49.5	4,295	LocalScore ↗	512+128	llamafile 0.9.2	Details →
Meta Llama 3.1 8B Instruct	Q4_K - Medium	45.8	4,682	LocalScore ↗	512+128	llamafile 0.9.2	Details →
Llama 3.2 1B Instruct	Q4_K - Medium	45.7	11,191	LocalScore ↗	512+128	llamafile 0.9.3	Details →
Llama 3.2 1B Instruct	Q4_K - Medium	45.3	10,888	LocalScore ↗	512+128	llamafile 0.9.3	Details →
Llama 3.2 1B Instruct	Q4_K - Medium	45.1	10,856	LocalScore ↗	512+128	llamafile 0.9.3	Details →
Llama 3.2 1B Instruct	Q4_K - Medium	45.0	11,332	LocalScore ↗	512+128	llamafile 0.9.3	Details →
Llama 3.2 1B Instruct	Q4_K - Medium	43.5	10,900	LocalScore ↗	512+128	llamafile 0.9.3	Details →
Meta Llama 3.1 8B Instruct	Q4_K - Medium	43.2	6,307	LocalScore ↗	512+128	llamafile 0.9.2	Details →
Llama 3.2 1B Instruct	Q4_K - Medium	41.5	10,392	LocalScore ↗	512+128	llamafile 0.9.3	Details →
Llama 3.2 1B Instruct	Q4_K - Medium	41.4	10,144	LocalScore ↗	512+128	llamafile 0.9.3	Details →
Qwen2.5 14B Instruct	Q4_K - Medium	27.4	2,671	LocalScore ↗	512+128	llamafile 0.9.2	Details →
Qwen2.5 14B Instruct	Q4_K - Medium	26.1	2,626	LocalScore ↗	512+128	llamafile 0.9.2	Details →
Qwen2.5 14B Instruct	Q4_K - Medium	24.3	2,451	LocalScore ↗	512+128	llamafile 0.9.2	Details →

Ollama

No measured Ollama results yet. Submit yours →

MLX

No measured MLX results yet. Submit yours →

Includes results from LocalScore, a Mozilla Builders project by cjpais, mirrored in the cn0303/local-llm-speed-benchmarks dataset. LocalScore runs use llamafile, which is built on llama.cpp.