Hardware /
RTX 3060
45measured results
214best gen tok/s*
12 GBreported memory
*Headline excludes tiny/test models (under 0.6B params or TinyStories-style). Open in rig checker →
llama.cpp
| Model | Quant | Gen tok/s | Prompt tok/s | Source | Context tested | Build | |
|---|---|---|---|---|---|---|---|
| Llama 3.2 1B Instruct | Q4_K - Medium | 214 | 6,358 | LocalScore ↗ | 512+128 | llamafile 0.9.2 | Details → |
| Llama 3.2 1B Instruct | Q4_K - Medium | 211 | 6,479 | LocalScore ↗ | 512+128 | llamafile 0.9.2 | Details → |
| Llama 3.2 1B Instruct | Q4_K - Medium | 211 | 6,061 | LocalScore ↗ | 512+128 | llamafile 0.9.2 | Details → |
| Llama 3.2 1B Instruct | Q4_K - Medium | 206 | 6,119 | LocalScore ↗ | 512+128 | llamafile 0.9.2 | Details → |
| Llama 3.2 1B Instruct | Q4_K - Medium | 202 | 5,800 | LocalScore ↗ | 512+128 | llamafile 0.9.2 | Details → |
| Llama 3.2 1B Instruct | Q4_K - Medium | 201 | 6,022 | LocalScore ↗ | 512+128 | llamafile 0.9.2 | Details → |
| Llama 3.2 1B Instruct | Q4_K - Medium | 197 | 6,058 | LocalScore ↗ | 512+128 | llamafile 0.9.2 | Details → |
| Llama 3.2 1B Instruct | Q4_K - Medium | 169 | 6,019 | LocalScore ↗ | 512+128 | llamafile 0.9.2 | Details → |
| Llama 3.2 1B Instruct | Q4_K - Medium | 157 | 6,254 | LocalScore ↗ | 512+128 | llamafile 0.9.3 | Details → |
| Llama 2 7B | Q4_0 | 80.6 | 2,013 | llama.cpp (Vulkan) ↗ | 512+128 | llama.cpp Vulkan @92c0b38 — coopmat2 | Details → |
| Llama 2 7B | Q4_0 | 76.9 | 2,408 | llama.cpp (CUDA) ↗ | 512+128 | llama.cpp CUDA @baa9255 | Details → |
| Llama 2 7B | Q4_0 | 75.9 | 1,816 | llama.cpp (Vulkan) ↗ | 512+128 | llama.cpp Vulkan @92c0b38 — coopmat2 | Details → |
| Llama 2 7B | Q4_0 | 75.6 | 2,138 | llama.cpp (CUDA) ↗ | 512+128 | llama.cpp CUDA @baa9255 | Details → |
| Llama 3.2 1B Instruct | Q4_K - Medium | 70.8 | 5,472 | LocalScore ↗ | 4,352 | llamafile 0.9.3 | Details → |
| Gemma 3 4b It Qat | Q4_0 | 65.1 | 2,981 | LocalScore ↗ | 512+128 | llamafile 0.9.2 | Details → |
| Meta Llama 3.1 8B Instruct | Q4_K - Medium | 57.5 | 1,545 | LocalScore ↗ | 512+128 | llamafile 0.9.2 | Details → |
| Meta Llama 3.1 8B Instruct | Q4_K - Medium | 53.1 | 1,487 | LocalScore ↗ | 512+128 | llamafile 0.9.2 | Details → |
| Meta Llama 3.1 8B Instruct | Q4_K - Medium | 53.0 | 1,506 | LocalScore ↗ | 512+128 | llamafile 0.9.2 | Details → |
| Meta Llama 3.1 8B Instruct | Q4_K - Medium | 52.8 | 1,503 | LocalScore ↗ | 512+128 | llamafile 0.9.2 | Details → |
| Meta Llama 3.1 8B Instruct | Q4_K - Medium | 52.4 | 1,478 | LocalScore ↗ | 512+128 | llamafile 0.9.2 | Details → |
| Meta Llama 3.1 8B Instruct | Q4_K - Medium | 52.2 | 1,429 | LocalScore ↗ | 512+128 | llamafile 0.9.2 | Details → |
| Meta Llama 3.1 8B Instruct | Q4_K - Medium | 52.0 | 1,482 | LocalScore ↗ | 512+128 | llamafile 0.9.2 | Details → |
| Meta Llama 3.1 8B Instruct | Q4_K - Medium | 49.7 | 1,400 | LocalScore ↗ | 512+128 | llamafile 0.9.2 | Details → |
| Meta Llama 3.1 8B Instruct | Q4_K - Medium | 49.6 | 1,555 | LocalScore ↗ | 512+128 | llamafile 0.9.3 | Details → |
| Meta Llama 3.1 8B Instruct | Q4_K - Medium | 49.5 | 1,515 | LocalScore ↗ | 512+128 | llamafile 0.9.2 | Details → |
| gemma-2-9b-it | Q4_K - Medium | 36.4 | 1,217 | LocalScore ↗ | 512+128 | llamafile 0.9.2 | Details → |
| Qwen2.5 14B Instruct | Q4_K - Medium | 32.8 | 846 | LocalScore ↗ | 512+128 | llamafile 0.9.2 | Details → |
| Gemma 3 12b It Qat | Q4_0 | 30.4 | 1,075 | LocalScore ↗ | 512+128 | llamafile 0.9.2 | Details → |
| Qwen2.5 14B Instruct | Q4_K - Medium | 29.7 | 835 | LocalScore ↗ | 512+128 | llamafile 0.9.2 | Details → |
| Qwen2.5 14B Instruct | Q4_K - Medium | 29.5 | 802 | LocalScore ↗ | 512+128 | llamafile 0.9.2 | Details → |
| Amoral Gemma3 12B | Q4_K - Medium | 29.5 | 1,004 | LocalScore ↗ | 512+128 | llamafile 0.9.2 | Details → |
| Qwen2.5 14B Instruct | Q4_K - Medium | 29.4 | 832 | LocalScore ↗ | 4,352 | llamafile 0.9.2 | Details → |
| Qwen2.5 14B Instruct | Q4_K - Medium | 29.2 | 799 | LocalScore ↗ | 512+128 | llamafile 0.9.2 | Details → |
| Qwen2.5 14B Instruct | Q4_K - Medium | 29.1 | 713 | LocalScore ↗ | 512+128 | llamafile 0.9.2 | Details → |
| Qwen2.5 14B Instruct | Q4_K - Medium | 28.6 | 793 | LocalScore ↗ | 512+128 | llamafile 0.9.2 | Details → |
| Qwen2.5 14B Instruct | Q4_K - Medium | 28.1 | 832 | LocalScore ↗ | 512+128 | llamafile 0.9.2 | Details → |
| Qwen2.5 14B Instruct | Q4_K - Medium | 27.6 | 752 | LocalScore ↗ | 512+128 | llamafile 0.9.2 | Details → |
| Qwen2.5 14B Instruct | Q4_K - Medium | 27.4 | 783 | LocalScore ↗ | 512+128 | llamafile 0.9.3 | Details → |
| Qwen2.5 14B Instruct | Q4_K - Medium | 27.4 | 761 | LocalScore ↗ | 512+128 | llamafile 0.9.2 | Details → |
| Qwen2.5 14B Instruct | Q4_K - Medium | 27.1 | 798 | LocalScore ↗ | 512+128 | llamafile 0.9.2 | Details → |
| Qwen2.5 14B Instruct | Q4_K - Medium | 26.9 | 791 | LocalScore ↗ | 512+128 | llamafile 0.9.2 | Details → |
| Qwen2.5 14B Instruct | Q4_K - Medium | 26.0 | 803 | LocalScore ↗ | 512+128 | llamafile 0.9.2 | Details → |
| Gemma 3 4b It | Q6_K | 23.2 | 1,063 | LocalScore ↗ | 512+128 | llamafile 0.9.2 | Details → |
| Qwen2.5 14B Instruct | Q4_K - Medium | 13.4 | 357 | LocalScore ↗ | 512+128 | llamafile 0.9.2 | Details → |
| Qwen2.5 14B Instruct | Q4_K - Medium | 9.5 | 547 | LocalScore ↗ | 4,352 | llamafile 0.9.3 | Details → |
Ollama
No measured Ollama results yet. Submit yours →
MLX
No measured MLX results yet. Submit yours →
Includes results from LocalScore, a Mozilla Builders project by cjpais, mirrored in the cn0303/local-llm-speed-benchmarks dataset. LocalScore runs use llamafile, which is built on llama.cpp.