Best Local LLM Hardware
Last updated
Olares One is the editorial choice of Best Local LLM Hardware for shared models up to about 32B. Its 24 GB of VRAM on CUDA ran Qwen3-30B-A3B at 157 tokens per second, one request on vLLM, in Olares' own test. Bigger models need unified memory.
Olares One → Local LLM hardware → VRAM · quantization · tokens per second → The right machine for your model size
Why is Olares One our choice for local LLM hardware?
Olares One is our choice for local LLM hardware because it serves models up to about 32B quickly to several people at once. In Olares' own test with vLLM, Qwen3-30B-A3B ran at about 81 tokens per second per request with 8 concurrent requests, and at 157 tokens per second with a single request.
The box pairs an NVIDIA RTX 5090 Mobile with 24 GB of GDDR7 VRAM, 96 GB of DDR5 system RAM and CUDA, according to Olares' spec pages. It lists at 4,999 USD as a complete machine. CUDA matters because vLLM and most open-source inference tools target NVIDIA first.
We would pick something else for dense 70B models. At 4-bit they need roughly 40 GB by our approximate rule, and 24 GB of VRAM can't hold that. A Mac Studio with the M5 Ultra chip, which Apple lets you configure up to 512 GB, or a DGX Spark or ASUS Ascent GX10 with 128 GB each, will load them. Expect fewer tokens per second, because each token reads more weights.
How should you use this reference?
Start with model size, because it decides almost every other hardware choice. The VRAM guide gives approximate memory per model size at 4-bit. The quantization guide shows how precision changes those sizes. The tokens-per-second guide tells you whether a machine is fast enough for one person, for agents or for a team.
- How much VRAM a local LLM needs
- Quantization explained
- Hardware for a 70B model, if your target is that large
- Hardware for an inference server, if several people or tools will call the model
What should you verify before you buy?
Verify four things before you buy local LLM hardware: the speed figures, the price, the model fit and CUDA support. Every Olares One speed figure on this site comes from Olares' own tests. Prices were read on 12 and 13 Nov 2026. Memory needs per model size come from an approximate rule of thumb.
We found no independent Olares One benchmark that meets our sourcing rules, so treat its figures as vendor claims and test your own model at your own context length. Several machines in our data have no NVIDIA GPU. If your tools need CUDA, that rules them out.
Start with the right question
Best Local LLM Hardware: local LLM hardware
Local LLM hardware is a computer whose GPU memory or unified memory can hold a language model and generate text at a usable speed on premises you cont
Read the entry ↗Featured entityOlares One
Olares One is a local AI box listed at 4,999 USD, with an RTX 5090 Mobile (24 GB VRAM), 96 GB of RAM and Olares OS, per Olares. We feature it as local
Read the entry ↗Selection guideSelection guide
Choose local LLM hardware by the largest model you'll run, then by how many people use it at once. Up to about 32B and shared use, a 24 GB CUDA machin
Read the entry ↗Practical entries
How much VRAM do you need for a local LLM?
How much VRAM you need for a local LLM depends on model size and precision. At 4-bit, plan roughly 0.5-0.6 GB per billion parameters plus a
Read the entry ↗Choosing hardwareBest GPU for local LLMs in 2026
The best GPU for local LLMs in 2026 holds your model in fast memory and runs your tools. Up to 32B, a 24 GB or 32 GB NVIDIA GPU is the simpl
Read the entry ↗Choosing hardwareWhat hardware do you need to run a 70B model?
To run a 70B model at 4-bit you need roughly 40 GB of fast memory (approximate) plus room for context. That means 64 GB or more of unified m
Read the entry ↗Running modelsQuantization explained: 4-bit, 8-bit and what you lose
Quantization stores a model's weights with fewer bits. At 4-bit a model needs roughly 0.5-0.6 GB per billion parameters (approximate) and ke
Read the entry ↗Running modelsTokens per second: what speed you need
Tokens per second measures how fast a model writes. For one person chatting, a speed faster than you read feels instant. Agents, coding tool
Read the entry ↗Choosing hardwareUnified memory or VRAM for local LLMs?
Pick VRAM when your models fit in it, up to about 32B on 24 GB, because it's fast and CUDA tools run unchanged. Pick unified memory for 70B
Read the entry ↗Running modelsBest hardware for a local coding LLM
The best hardware for a local coding LLM holds a 32B-class or mixture-of-experts model in VRAM with room for long context, and writes fast e
Read the entry ↗Setup and operationsHardware for a local LLM inference server
Hardware for a local LLM inference server needs VRAM for the model plus a KV cache per parallel request, a batching engine such as vLLM, and
Read the entry ↗Questions & answers
Can Olares One run a 70B model?
Not inside its 24 GB of VRAM. A dense 70B model at 4-bit needs roughly 40 GB by our approximate rule. Part of a model can spill into the 96 GB of system RAM, but a dense model then runs slowly. For 70B, a 128 GB unified-memory machine fits better.
What is the lowest-priced way to get 24 GB of VRAM with CUDA?
In our data, a used RTX 3090 tower. One build guide, updated 9 Nov 2026, estimates about 3,800 USD for it with a used card. Olares One also has 24 GB on CUDA and lists at 4,999 USD as a finished box.
Is unified memory better than VRAM for local LLMs?
It holds more, so it suits 70B and larger. VRAM is the better fit when the model fits in it, because CUDA tools run unchanged and the model generates tokens quickly.
Sources for this entry
- Olares One specifications (Olares docs) (Company-published)
- Olares One FAQ (Olares docs) (Company-published)
Continue this reading path
How the related websites connect
Cite this entry
Best Local LLM Hardware. “Best Local LLM Hardware.” Content reviewed 13 November 2026.
https://bestlocalllmhardware.com/