Qwen2.5 14B Instruct on RTX 3060

32.8best gen tok/s

846best prompt tok/s

4,352max context tested

16results

Measured · all numbers below come from the linked sources

llama.cpp

Quant	Gen tok/s	Prompt tok/s	Source	Context tested	TTFT	Build
Q4_K - Medium	32.8	846	LocalScore ↗	512+128	1,527 ms	llamafile 0.9.2
Q4_K - Medium	29.7	835	LocalScore ↗	512+128	1,560 ms	llamafile 0.9.2
Q4_K - Medium	29.5	802	LocalScore ↗	512+128	1,624 ms	llamafile 0.9.2
Q4_K - Medium	29.4	832	LocalScore ↗	4,352	1,577 ms	llamafile 0.9.2
Q4_K - Medium	29.2	799	LocalScore ↗	512+128	1,648 ms	llamafile 0.9.2
Q4_K - Medium	29.1	713	LocalScore ↗	512+128	2,348 ms	llamafile 0.9.2
Q4_K - Medium	28.6	793	LocalScore ↗	512+128	1,662 ms	llamafile 0.9.2
Q4_K - Medium	28.1	832	LocalScore ↗	512+128	1,582 ms	llamafile 0.9.2
Q4_K - Medium	27.6	752	LocalScore ↗	512+128	1,719 ms	llamafile 0.9.2
Q4_K - Medium	27.4	783	LocalScore ↗	512+128	1,661 ms	llamafile 0.9.3
Q4_K - Medium	27.4	761	LocalScore ↗	512+128	1,731 ms	llamafile 0.9.2
Q4_K - Medium	27.1	798	LocalScore ↗	512+128	1,631 ms	llamafile 0.9.2
Q4_K - Medium	26.9	791	LocalScore ↗	512+128	1,625 ms	llamafile 0.9.2
Q4_K - Medium	26.0	803	LocalScore ↗	512+128	1,614 ms	llamafile 0.9.2
Q4_K - Medium	13.4	357	LocalScore ↗	512+128	5,601 ms	llamafile 0.9.2
Q4_K - Medium	9.5	547	LocalScore ↗	4,352	2,307 ms	llamafile 0.9.3

Ollama

No measured Ollama results yet. Submit yours →

MLX

No measured MLX results yet. Submit yours →

"Context tested" is the largest prompt+generation length actually benchmarked (tokens), not a memory-fit estimate.

What else can RTX 3060 run? →

Includes results from LocalScore, a Mozilla Builders project by cjpais, mirrored in the cn0303/local-llm-speed-benchmarks dataset. LocalScore runs use llamafile, which is built on llama.cpp.