RTX A5000

15measured results

313best gen tok/s*

24 GBreported memory

*Headline excludes tiny/test models (under 0.6B params or TinyStories-style). Open in rig checker →

llama.cpp

Model	Quant	Gen tok/s	Prompt tok/s	Source	Context tested	Build
Llama 3.2 1B Instruct	Q4_K - Medium	313	11,835	LocalScore ↗	512+128	llamafile 0.9.2	Details →
Llama 3.2 1B Instruct	Q4_K - Medium	310	12,253	LocalScore ↗	512+128	llamafile 0.9.2	Details →
Llama 3.2 1B Instruct	Q4_K - Medium	307	12,325	LocalScore ↗	512+128	llamafile 0.9.2	Details →
Llama 3.2 1B Instruct	Q6_K	273	11,451	LocalScore ↗	512+128	llamafile 0.9.2	Details →
Llama 2 7B	Q4_0	140	4,071	llama.cpp (Vulkan) ↗	512+128	llama.cpp Vulkan @4ae88d0 — coopmat2	Details →
Llama 2 7B	Q4_0	140	3,642	llama.cpp (Vulkan) ↗	512+128	llama.cpp Vulkan @4ae88d0 — coopmat2	Details →
Llama 2 7B	Q4_0	136	4,552	llama.cpp (CUDA) ↗	512+128	llama.cpp CUDA @e5155e6	Details →
Llama 2 7B	Q4_0	130	4,028	llama.cpp (CUDA) ↗	512+128	llama.cpp CUDA @e5155e6	Details →
Meta Llama 3.1 8B Instruct	Q4_K - Medium	89.9	2,890	LocalScore ↗	512+128	llamafile 0.9.2	Details →
Meta Llama 3.1 8B Instruct	Q4_K - Medium	88.3	2,891	LocalScore ↗	512+128	llamafile 0.9.2	Details →
Ministral 8B Instruct 2410 TEST	Q4_K - Medium	86.6	2,817	LocalScore ↗	512+128	llamafile 0.9.2	Details →
Mistral Nemo Instruct 2407	Q4_K - Medium	63.1	1,959	LocalScore ↗	512+128	llamafile 0.9.2	Details →
Qwen2.5 14B Instruct	Q4_K - Medium	50.9	1,524	LocalScore ↗	512+128	llamafile 0.9.2	Details →
Qwen2.5 14B Instruct	Q4_K - Medium	50.0	1,541	LocalScore ↗	512+128	llamafile 0.9.2	Details →
Mistral Small 24B Instruct 2501	Q4_K - Medium	38.8	1,116	LocalScore ↗	512+128	llamafile 0.9.2	Details →

Ollama

No measured Ollama results yet. Submit yours →

MLX

No measured MLX results yet. Submit yours →

Includes results from LocalScore, a Mozilla Builders project by cjpais, mirrored in the cn0303/local-llm-speed-benchmarks dataset. LocalScore runs use llamafile, which is built on llama.cpp.