Meta Llama 3.1 8B Instruct on RTX 3090

117best gen tok/s

3,991best prompt tok/s

4,352max context tested

16results

Measured · all numbers below come from the linked sources

llama.cpp

Quant	Gen tok/s	Prompt tok/s	Source	Context tested	TTFT	Build
Q4_K - Medium	117	3,895	LocalScore ↗	4,352	326 ms	llamafile 0.9.2
Q4_K - Medium	117	3,926	LocalScore ↗	4,352	325 ms	llamafile 0.9.2
Q4_K - Medium	113	3,991	LocalScore ↗	512+128	326 ms	llamafile 0.9.2
Q4_K - Medium	110	3,895	LocalScore ↗	512+128	332 ms	llamafile 0.9.2
Q4_K - Medium	110	3,898	LocalScore ↗	512+128	330 ms	llamafile 0.9.2
Q4_K - Medium	109	3,858	LocalScore ↗	512+128	335 ms	llamafile 0.9.2
Q4_K - Medium	103	3,552	LocalScore ↗	512+128	358 ms	llamafile 0.9.1
Q4_K - Medium	103	3,637	LocalScore ↗	512+128	352 ms	llamafile 0.9.2
Q4_K - Medium	103	3,511	LocalScore ↗	4,352	369 ms	llamafile 0.9.2
Q4_K - Medium	101	3,454	LocalScore ↗	512+128	368 ms	llamafile 0.9.2
Q4_K - Medium	101	3,307	LocalScore ↗	4,352	383 ms	llamafile 0.9.2
Q4_K - Medium	85.4	3,876	LocalScore ↗	512+128	329 ms	llamafile 0.9.2
Q4_K - Medium	83.1	3,641	LocalScore ↗	512+128	347 ms	llamafile 0.9.2
Q4_K - Medium	79.7	3,363	LocalScore ↗	4,352	377 ms	llamafile 0.9.3
Q4_K - Medium	65.4	2,251	LocalScore ↗	512+128	2,362 ms	llamafile 0.9.2
Q4_K - Medium	25.1	2,540	LocalScore ↗	4,352	508 ms	llamafile 0.9.3

Ollama

No measured Ollama results yet. Submit yours →

MLX

No measured MLX results yet. Submit yours →

"Context tested" is the largest prompt+generation length actually benchmarked (tokens), not a memory-fit estimate.

What else can RTX 3090 run? →

Includes results from LocalScore, a Mozilla Builders project by cjpais, mirrored in the cn0303/local-llm-speed-benchmarks dataset. LocalScore runs use llamafile, which is built on llama.cpp.