Hardware /
RTX 5070 Ti
47measured results
260best gen tok/s*
16 GBreported memory
*Headline excludes tiny/test models (under 0.6B params or TinyStories-style). Open in rig checker →
llama.cpp
| Model | Quant | Gen tok/s | Prompt tok/s | Source | Context tested | Build | |
|---|---|---|---|---|---|---|---|
| Llama 3.2 1B Instruct | Q4_K - Medium | 260 | 15,534 | LocalScore ↗ | 512+128 | llamafile 0.9.2 | Details → |
| Llama 3.2 1B Instruct | Q4_K - Medium | 254 | 15,705 | LocalScore ↗ | 512+128 | llamafile 0.9.2 | Details → |
| Llama 3.2 1B Instruct | Q4_K - Medium | 240 | 16,482 | LocalScore ↗ | 512+128 | llamafile 0.9.2 | Details → |
| Llama 3.2 1B Instruct | Q4_K - Medium | 240 | 16,447 | LocalScore ↗ | 512+128 | llamafile 0.9.2 | Details → |
| Llama 3.2 1B Instruct | Q4_K - Medium | 238 | 16,091 | LocalScore ↗ | 512+128 | llamafile 0.9.2 | Details → |
| Llama 3.2 1B Instruct | Q4_K - Medium | 238 | 16,125 | LocalScore ↗ | 512+128 | llamafile 0.9.2 | Details → |
| Llama 2 7B | Q4_0 | 182 | 8,420 | llama.cpp (CUDA) ↗ | 512+128 | llama.cpp CUDA @933414c | Details → |
| Llama 2 7B | Q4_0 | 177 | 6,952 | llama.cpp (CUDA) ↗ | 512+128 | llama.cpp CUDA @933414c | Details → |
| Llama 2 7B | Q4_0 | 176 | 7,568 | llama.cpp (Vulkan) ↗ | 512+128 | llama.cpp Vulkan @0d0764d — coopmat2 | Details → |
| Qwen1.5-0.5B-Chat-AWQ-fp16 | Q4_K - Medium | 172 | 16,250 | LocalScore ↗ | 4,352 | llamafile 0.9.3 | Details → |
| Llama 2 7B | Q4_0 | 136 | 6,214 | llama.cpp (Vulkan) ↗ | 512+128 | llama.cpp Vulkan @d13d0f6 — coopmat2 | Details → |
| Llama 3.2 1B Instruct | Q4_K - Medium | 123 | 14,483 | LocalScore ↗ | 512+128 | llamafile 0.9.2 | Details → |
| Llama 3.2 1B Instruct | Q4_K - Medium | 118 | 13,485 | LocalScore ↗ | 512+128 | llamafile 0.9.2 | Details → |
| Llama 3.2 1B Instruct | Q4_K - Medium | 110 | 10,769 | LocalScore ↗ | 512+128 | llamafile 0.9.3 | Details → |
| Llama 3.2 1B Instruct | Q4_K - Medium | 107 | 13,136 | LocalScore ↗ | 512+128 | llamafile 0.9.2 | Details → |
| Llama 3.2 1B Instruct | Q4_K - Medium | 106 | 10,130 | LocalScore ↗ | 512+128 | llamafile 0.9.2 | Details → |
| Llama 3.2 1B Instruct | Q4_K - Medium | 100 | 11,177 | LocalScore ↗ | 512+128 | llamafile 0.9.3 | Details → |
| Llama 3.2 1B Instruct | Q4_K - Medium | 98.3 | 9,569 | LocalScore ↗ | 512+128 | llamafile 0.9.3 | Details → |
| Meta Llama 3.1 8B Instruct | Q4_K - Medium | 91.8 | 3,860 | LocalScore ↗ | 512+128 | llamafile 0.9.2 | Details → |
| Llama 3.2 1B Instruct | Q4_K - Medium | 91.0 | 9,623 | LocalScore ↗ | 512+128 | llamafile 0.9.3 | Details → |
| Llama 3.2 1B Instruct | Q4_K - Medium | 87.8 | 7,272 | LocalScore ↗ | 512+128 | llamafile 0.9.3 | Details → |
| Llama 3.2 1B Instruct | Q4_K - Medium | 86.8 | 7,372 | LocalScore ↗ | 512+128 | llamafile 0.9.3 | Details → |
| Meta Llama 3.1 8B Instruct | Q4_K - Medium | 79.8 | 4,323 | LocalScore ↗ | 512+128 | llamafile 0.9.2 | Details → |
| Llama 3.2 1B Instruct | Q4_K - Medium | 53.4 | 9,413 | LocalScore ↗ | 4,352 | llamafile 0.9.3 | Details → |
| Meta Llama 3.1 8B Instruct | Q4_K - Medium | 48.6 | 3,827 | LocalScore ↗ | 512+128 | llamafile 0.9.2 | Details → |
| Meta Llama 3.1 8B Instruct | Q4_K - Medium | 48.6 | 4,023 | LocalScore ↗ | 512+128 | llamafile 0.9.3 | Details → |
| Llama 3.2 1B Instruct | Q4_K - Medium | 47.7 | 9,439 | LocalScore ↗ | 4,352 | llamafile 0.9.3 | Details → |
| Qwen2.5 14B Instruct | Q4_K - Medium | 47.7 | 2,317 | LocalScore ↗ | 512+128 | llamafile 0.9.2 | Details → |
| Llama 3.2 1B Instruct | Q4_K - Medium | 46.8 | 9,444 | LocalScore ↗ | 4,352 | llamafile 0.9.3 | Details → |
| Llama 3.2 1B Instruct | Q4_K - Medium | 44.0 | 8,711 | LocalScore ↗ | 512+128 | llamafile 0.9.3 | Details → |
| Meta Llama 3.1 8B Instruct | Q4_K - Medium | 41.9 | 2,617 | LocalScore ↗ | 512+128 | llamafile 0.9.2 | Details → |
| Qwen2.5 14B Instruct | Q4_K - Medium | 41.2 | 2,490 | LocalScore ↗ | 512+128 | llamafile 0.9.3 | Details → |
| Qwen2.5 14B Instruct | Q4_K - Medium | 36.5 | 2,367 | LocalScore ↗ | 512+128 | llamafile 0.9.3 | Details → |
| Meta Llama 3.1 8B Instruct | Q4_K - Medium | 35.4 | 1,961 | LocalScore ↗ | 512+128 | llamafile 0.9.3 | Details → |
| Meta Llama 3.1 8B Instruct | Q4_K - Medium | 33.7 | 1,951 | LocalScore ↗ | 512+128 | llamafile 0.9.3 | Details → |
| Meta Llama 3.1 8B Instruct | Q4_K - Medium | 33.3 | 1,955 | LocalScore ↗ | 512+128 | llamafile 0.9.3 | Details → |
| Meta Llama 3.1 8B Instruct | Q4_K - Medium | 33.1 | 2,475 | LocalScore ↗ | 512+128 | llamafile 0.9.3 | Details → |
| Qwen2.5 14B Instruct | Q4_K - Medium | 31.1 | 2,195 | LocalScore ↗ | 512+128 | llamafile 0.9.3 | Details → |
| Qwen2.5 14B Instruct | Q4_K - Medium | 27.6 | 2,261 | LocalScore ↗ | 512+128 | llamafile 0.9.2 | Details → |
| Qwen2.5 14B Instruct | Q4_K - Medium | 26.9 | 2,262 | LocalScore ↗ | 512+128 | llamafile 0.9.2 | Details → |
| Meta Llama 3.1 8B Instruct | Q4_K - Medium | 24.0 | 3,083 | LocalScore ↗ | 4,352 | llamafile 0.9.3 | Details → |
| Qwen2.5 14B Instruct | Q4_K - Medium | 23.2 | 1,482 | LocalScore ↗ | 512+128 | llamafile 0.9.2 | Details → |
| Qwen2.5 14B Instruct | Q4_K - Medium | 20.9 | 1,129 | LocalScore ↗ | 512+128 | llamafile 0.9.3 | Details → |
| Qwen2.5 14B Instruct | Q4_K - Medium | 20.0 | 1,127 | LocalScore ↗ | 512+128 | llamafile 0.9.3 | Details → |
| Qwen2.5 14B Instruct | Q4_K - Medium | 20.0 | 1,128 | LocalScore ↗ | 512+128 | llamafile 0.9.3 | Details → |
| Qwen2.5 14B Instruct | Q4_K - Medium | 14.2 | 1,740 | LocalScore ↗ | 4,352 | llamafile 0.9.3 | Details → |
| Qwen2.5 14B Instruct | Q4_K - Medium | 12.6 | 1,638 | LocalScore ↗ | 4,352 | llamafile 0.9.3 | Details → |
Ollama
No measured Ollama results yet. Submit yours →
MLX
No measured MLX results yet. Submit yours →
Includes results from LocalScore, a Mozilla Builders project by cjpais, mirrored in the cn0303/local-llm-speed-benchmarks dataset. LocalScore runs use llamafile, which is built on llama.cpp.