Running an LLM locally is no longer the preserve of data centres. Once compressed, an 8-billion-parameter model fits in the memory of a recent workstation, and a machine with 128 GB of memory can host a 70-billion-parameter model. The price: a little quality and a lot of memory. What you gain: your documents never leave the office.
Key points
- Memory decides everything. Compressed to about 4 bits, a model shrinks threefold: 4.58 GiB instead of 14.96 GiB for Llama 3.1 8B, according to the llama.cpp documentation.
- This compression, called quantisation, costs some quality, to be checked on your own cases.
- "Mixture of experts" models only activate part of themselves: gpt-oss-20b has 21 billion parameters, uses 3.6 billion for each word and fits in 16 GB of memory, according to OpenAI.
- Small "distilled" models inherit part of a large model's knowledge, not all of it.
- The engines are open source (MIT or Apache 2.0 licences), with one exception: LM Studio is a proprietary application.
Running an LLM locally: what your hardware allows
A language model is a large file of numbers, the parameters. To answer, the machine must keep them in memory and read them again for every word. Two questions are therefore enough to size it: how much memory, and how fast it can be read.
- Your existing workstation. With 16 to 32 GB of memory, a compressed 7- to 8-billion-parameter model runs: slowly on the processor alone, decently with a graphics card. Enough to summarise, rephrase or sort emails.
- A workstation with a graphics card. NVIDIA sells consumer cards with 16 GB of memory (GeForce RTX 5060 Ti) and up to 32 GB (GeForce RTX 5090). The model must fit in that memory to run fast.
- A unified-memory machine. On a recent Mac or a PC built on the AMD Ryzen AI Max+ 395, the processor and the graphics chip share the same memory: up to 64 GB on a Mac mini, 128 GB with the AMD processor, 512 GB on a Mac Studio. That is where 30- to 70-billion-parameter models live.
Recent PCs also have a chip dedicated to AI, the NPU: Microsoft advertises over 40 trillion operations per second for its Copilot+ PCs. It mostly serves small models built into the system. For a mid-sized assistant, I look at memory first.
What you lose in quality
Two losses, not to be confused. The first comes from compression. The researchers who published the GPTQ method brought a 175-billion-parameter model down to 3 or 4 bits per parameter with a loss of accuracy they describe as negligible. The llama.cpp authors point out, however, that a loss does exist and can be measured.
The second, larger one comes from size. DeepSeek publishes its own figures: on the same mathematics test (AIME 2024), its 1.5-billion-parameter distilled version solves 28.9% of the problems, the 32-billion version 72.6%.
My rule: a small local model for well-defined, repetitive tasks, and a larger model, local or external under contract, for hard reasoning. The tool comparison helps you choose the interface.
What stays with you
The model file is yours: it does not change without your consent, it does not vanish when a vendor retires a version, and your questions and documents stay on your machine. The strategic reasons are in my article on self-hosted AI.
Under the hood: for your IT person
Memory needed ≈ weights + KV cache + headroom. Weights in bytes = parameters × bits per weight / 8. KV cache per token = 2 (key and value) × layers × KV heads × head dimension × bytes per value. Example from the official Qwen3-8B configuration (8.2 billion parameters, 36 layers, 8 KV heads, dimension 128): 147,456 bytes per token in 16-bit, about 4.8 GB for a 32,768-token context. Weights in Q4_K_M (4.89 bits per weight) come to about 5 GB. Total: around 10 GB, which fits a 16 GB card; in BF16 the weights alone (16.4 GB) do not.
- Formats. GGUF is llama.cpp's single-file format, metadata included. Its k-quants (Q4_K_M, Q5_K_M, Q6_K) quantise in blocks with scale factors; Q8_0 is 8.5 bits per weight. On GPUs, GPTQ (second-order information, 3 to 4 bits) and AWQ (protect about 1% of salient weights, identified from activations) are served by vLLM. QLoRA's NF4 belongs to custom training.
- MoE. Qwen3-30B-A3B: 30.5 billion parameters in total, 3.3 billion activated, 8 of 128 experts per token. Memory is sized on the total (about 19 GB in Q4_K_M), compute on the active part. gpt-oss-20b ships quantised in MXFP4.
- Distillation. Hinton, Vinyals and Dean (2015) train a small model on the outputs of a large one. The DeepSeek-R1-Distill models (1.5 to 70 billion) are Qwen 2.5 and Llama 3 models fine-tuned on 800,000 samples generated by R1: each model's licence follows its base.
- Engines. llama.cpp (MIT, v1.0.0): CPU, Metal, CUDA, hybrid CPU and GPU execution for models larger than VRAM, MoE expert cache in host memory. Ollama (MIT): the simplest wrapper, but it also offers cloud models, so check. vLLM (Apache 2.0): multi-user server, PagedAttention. MLX (MIT): Apple's framework for Apple silicon. LM Studio: proprietary application, internal business use allowed by its terms.
- Hardware. Generation speed follows memory bandwidth: 1.2 TB/s on the Mac Studio M5 Ultra, 614 GB/s on M5 Max. The AMD Ryzen AI Max+ 395 NPU is rated at 50 TOPS, out of 126 TOPS overall.
What I have learned
In my home automation infrastructure, speech recognition (Whisper) and speech synthesis (Piper) run locally, offline, on a server with a 12-core AMD Ryzen processor and 64 GB of memory: about 800 ms to transcribe, 400 ms to answer. Reasoning, on the other hand, goes through an external API.
On DataTerra, the vision models run on the processor, without a graphics card, most of them exported to ONNX. My takeaway: specialised models run very well on ordinary hardware. It is general-purpose reasoning that demands memory, and that is where you must sort what stays in-house from what goes out, counting the cost of APIs too.
Where to start
1. Write down twenty real cases
Emails to sort, contracts to summarise, questions from your clients, with the expected answer. Without this list, no test means anything.
2. Try it on an existing machine
A compressed 7- to 8-billion-parameter model, run with Ollama on a 16 GB workstation, is enough to see what works and what does not.
3. Size the hardware afterwards
If the small model fails on your cases, try a larger or mixture-of-experts model, then calculate the memory needed before choosing a graphics card or a unified-memory machine.
Frequently asked questions
Is a laptop enough?
For personal use with a small model, often yes. For a team querying the same model all day, you need a dedicated machine.
Do I need an NVIDIA card?
No. llama.cpp runs on processors, on Apple chips and on cards from several manufacturers; vLLM also supports AMD and Intel cards.
Is a local model as good as a large online model?
Not for everything: on a well-defined task, often yes; for complex reasoning, the gap is real. Combine both depending on how sensitive the data is.
Going further
I help you choose the model and hardware that suit your cases, then install everything on your premises: see sovereign AI and infrastructure, or book a 20-minute discovery call.
Sources
- GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers, Frantar, Ashkboos, Hoefler, Alistarh, arXiv, 31 October 2022 (ICLR 2023)
- AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration, Lin et al., MIT, arXiv, 1 June 2023 (MLSys 2024)
- QLoRA: Efficient Finetuning of Quantized LLMs, Dettmers, Pagnoni, Holtzman, Zettlemoyer, arXiv, 23 May 2023
- Distilling the Knowledge in a Neural Network, Hinton, Vinyals, Dean, arXiv, 9 March 2015
- llama.cpp, ggml-org, GitHub, MIT licence, release v1.0.0 of 13 November 2026
- llama.cpp: quantize (sizes and bits per weight), ggml-org, GitHub, accessed 15 November 2026
- GGUF, ggml-org, GitHub, accessed 15 November 2026
- Ollama, Ollama, GitHub, MIT licence, release v0.40.16 of 9 November 2026
- vLLM, vLLM project, GitHub, Apache 2.0 licence, release v0.32.0 of 18 October 2026
- MLX, Apple (ml-explore), GitHub, MIT licence, release v0.32.4 of 14 October 2026
- LM Studio App Terms of Service, Element Labs, version of 23 August 2026
- gpt-oss-20b (model card), OpenAI, Hugging Face, 4 August 2025
- Qwen3-30B-A3B (model card), Qwen, Hugging Face, 27 April 2025
- Qwen3-8B (config.json), Qwen, Hugging Face, accessed 15 November 2026
- DeepSeek-R1 (model card), DeepSeek, Hugging Face, 20 January 2025
- GeForce RTX 5090, NVIDIA, accessed 15 November 2026
- GeForce RTX 5060 Family, NVIDIA, accessed 15 November 2026
- Mac Studio - Technical Specifications, Apple, accessed 15 November 2026
- Mac mini - Technical Specifications, Apple, accessed 15 November 2026
- AMD Ryzen AI Max+ 395, AMD, accessed 15 November 2026
- Copilot+ PCs, Microsoft, accessed 15 November 2026
Cover photo: Nick Stathas / Wikimedia Commons, CC BY-SA 4.0 licence.
Tags: AI, local LLM, quantisation, hardware, open source