Meta Llama 3.1 8B Instruct on DGX Spark

37.1best gen tok/s

2,110best prompt tok/s

4,352max context tested

10results

Measured · all numbers below come from the linked sources

llama.cpp

Quant	Gen tok/s	Prompt tok/s	Source	Context tested	TTFT	Build
Q4_K - Medium	37.1	2,110	LocalScore ↗	512+128	657 ms	llamafile 0.9.2
Q4_K - Medium	36.6	2,060	LocalScore ↗	512+128	670 ms	llamafile 0.9.2
Q4_K - Medium	36.5	2,102	LocalScore ↗	512+128	656 ms	llamafile 0.9.2
Q4_K - Medium	36.5	2,106	LocalScore ↗	4,352	658 ms	llamafile 0.9.2
Q4_K - Medium	34.9	2,027	LocalScore ↗	512+128	681 ms	llamafile 0.9.2
Q4_K - Medium	34.8	2,042	LocalScore ↗	512+128	676 ms	llamafile 0.9.2
Q4_K - Medium	33.2	2,035	LocalScore ↗	512+128	679 ms	llamafile 0.9.2
Q4_K - Medium	33.1	2,044	LocalScore ↗	512+128	676 ms	llamafile 0.9.2
Q4_K - Medium	30.8	1,831	LocalScore ↗	512+128	763 ms	llamafile 0.9.2
Q4_K - Medium	24.1	1,510	LocalScore ↗	512+128	1,005 ms	llamafile 0.9.2

Ollama

No measured Ollama results yet. Submit yours →

MLX

No measured MLX results yet. Submit yours →

"Context tested" is the largest prompt+generation length actually benchmarked (tokens), not a memory-fit estimate.

What else can DGX Spark run? →

Includes results from LocalScore, a Mozilla Builders project by cjpais, mirrored in the cn0303/local-llm-speed-benchmarks dataset. LocalScore runs use llamafile, which is built on llama.cpp.