Best Local LLM: A Guide to Running Large Language Models on Your Own Hardware

TLDR: Picking the best local LLM is a hardware-first decision. The model that fits your GPU or RAM determines your real option set, and benchmarks tell you less than a 30-minute evaluation on your actual task. This guide organizes the current open-weight landscape by hardware tier, explains what each major family is good at, and gives you a practical eval framework for making your own determination instead of trusting someone else's leaderboard run.

Start With Hardware, Not Hype

Benchmarks are measured on specific hardware configurations, specific prompts, and specific versions of each model. Your workload is different. The safest approach is to identify which models fit your hardware, then run them against representative samples of your real inputs, scoring outputs on criteria that matter for your application.

The main constraint is memory. The entire model must fit in GPU VRAM for fast inference. If a model exceeds your VRAM, some tools can offload layers to CPU RAM, but this usually drops throughput to CPU-inference speeds for the offloaded portion. Use the hardware tiers below to narrow your option set, then evaluate within it.

Hardware Tier 1: 8-16 GB VRAM (Laptop GPUs, Consumer Desktop)

This tier covers RTX 4060 (8 GB), RTX 4070 (12 GB), RTX 4080 (16 GB), and equivalent AMD cards, plus Apple M-series laptops with 16-24 GB unified memory.

Target model range: 7B to 14B parameter models at Q4_K_M quantization. A Q4_K_M 7B model uses roughly 4 GB; a Q4_K_M 14B model uses roughly 8-9 GB. This leaves headroom for context.

Qwen3-8B

Qwen3-8B (Alibaba Cloud, Apache 2.0 license) is one of the stronger models in this tier. Qwen3 was trained across 100+ languages and dialects. Qwen3-8B has a native context window of 32K tokens (up to 131K with YaRN scaling) and supports hybrid thinking mode: you can prompt it into a step-by-step reasoning mode for harder tasks or a fast direct-response mode for simpler ones. It runs well at Q4_K_M in Ollama on an 8 GB GPU. (Hugging Face model card, 2026-09-04)

ollama run qwen3:8b

Llama 3.1 8B

Meta's Llama 3.1 8B is an instruction-tuned model with 128K context length, multilingual support, and strong tool-use capability. It was pre-trained on roughly 15 trillion tokens. It is widely tested, well-supported across all major runtimes, and has a large base of fine-tunes and variants on Hugging Face. For general-purpose use at this tier it is a reliable baseline. Available at huggingface.co/meta-llama. (Ollama library, 2026-09-04)

Gemma 3 (4B and 12B)

Google DeepMind's Gemma 3 is a multimodal model family with text and image input. The 4B variant runs on anything with 8 GB; the 12B requires 12-16 GB at Q4. Gemma 3 has a 128K context window (the 1B variant has 32K) and was trained on 4 trillion tokens (4B) and 12 trillion tokens (12B). It is available through ai.google.dev/gemma. The 4B model is a good choice for RAM-constrained environments that also need multimodal input. (Hugging Face model card, 2026-09-04)

Mistral Small

Mistral AI's smaller dense models are consistently efficient relative to their size. Mistral-Small-3.1 is available on Ollama and Hugging Face. It handles instruction following well and is a reliable choice for summarization and structured output tasks at this tier.

Hardware Tier 2: 24 GB VRAM (Single Workstation GPU or 32 GB Apple Silicon)

This tier covers the RTX 4090 (24 GB), RTX 6000 Ada (48 GB), and Apple M2/M3/M4 Max with 32-96 GB unified memory. The 24 GB RTX 4090 is the most common "serious developer" GPU for local inference.

Target model range: Up to 30-32B parameter models at Q4_K_M. This tier opens up models that outperform the 7-8B class by a significant margin on complex reasoning and long-context tasks.

Qwen3-32B

Qwen3-32B is a dense 32.8B parameter model with a native context window of 32K tokens (up to 131K with YaRN scaling). Qwen claims Qwen3-4B rivals Qwen2.5-72B-Instruct performance, with each generation significantly more efficient per parameter. At Q4_K_M, Qwen3-32B fits within 24 GB with room for context. This is a strong general-purpose model for teams with a single high-end workstation GPU. (Hugging Face model card, 2026-09-04)

Llama 3.3 70B (with offloading) or Llama 3.1 70B

Meta's Llama 3.3 is a 70B text-only instruct-tuned model. At Q4_K_M, 70B models require roughly 40 GB, which exceeds 24 GB VRAM. Running them requires either CPU offloading (accepted throughput penalty), two GPUs, or a high-unified-memory Apple Silicon system. On a Mac with 64-96 GB unified memory, Llama 3.3 70B runs at reasonable throughput via Ollama's MLX backend.

Devstral Small 1.1 (coding)

Mistral's Devstral Small 1.1 is a 24B coding-focused model that fits exactly on a single RTX 4090 or a 32 GB Mac. It scores 53.6% on SWE-Bench Verified, which at publication was the highest score for any open-source model under a comparable evaluation scaffold. It is built for agentic software engineering tasks. Covered in more detail in the coding-specific article. Available at huggingface.co/mistralai/Devstral-Small-2507. (Hugging Face model card, 2026-09-04)

Gemma 4 (12B, 26B, 31B)

Google's Gemma 4 family includes 12B, 26B, and 31B models described as "frontier-level capabilities" for personal computers. The 12B fits comfortably; the 26B-31B variants require Q4 quantization to fit in 24 GB. Gemma 4 introduced multi-token prediction support, which Ollama uses to achieve up to 90% faster throughput on Apple Silicon according to Ollama's June 2026 blog post.

Hardware Tier 3: Multi-GPU Servers or High-Memory Single Cards

This tier covers dual-GPU or quad-GPU setups (e.g., 2x or 4x A100/H100 80 GB), workstations with 48 GB+ VRAM, and enterprise inference servers. This is where 70B dense models and large MoE models become practical for production throughput.

DeepSeek-V3

DeepSeek-V3 is a 671B total parameter MoE model with 37B parameters activated per token. It has a 128K context window and supports FP8 and BF16 precision. It was pre-trained on 14.8 trillion tokens. Despite the large parameter count, its activated parameter count (37B) means inference compute is closer to a 37B dense model than a 671B one. It requires significant multi-GPU hardware for local deployment; SGLang and vLLM are the recommended serving stacks. Available at huggingface.co/deepseek-ai/DeepSeek-V3 under MIT license.

Llama 3.1 405B

Meta's 405B model requires approximately 243 GB at Q4_K_M, meaning it needs several high-memory GPUs or H100 nodes. Meta described it as the first frontier-level open-weight model. It is relevant for teams doing distillation, synthetic data generation, or high-quality batch processing at scale who control the necessary infrastructure. (Ollama library, 2026-09-04)

Qwen3-235B-A22B

Qwen3's largest open-weight MoE model has 235B total parameters with 22B activated. Compared to dense 70B models, it delivers substantially better benchmark performance at a lower per-token compute cost. Requires distributed inference but is accessible with multi-A100 configurations.

Evaluating Models for Your Task

Public benchmarks (MMLU, HumanEval, MATH, etc.) measure aggregate capability across diverse tasks. Your application likely uses a much narrower slice of capabilities. A model that scores 2% higher on a general reasoning benchmark may score lower on your specific prompt format, domain vocabulary, or output structure requirements.

A practical evaluation protocol for selecting between two models:

1. Collect 20 to 50 representative inputs from your production workload (or simulated ones if it is a new application).
2. Define a scoring rubric: what does a good output look like? What are failure modes?
3. Run both models at your intended quantization level with the same system prompt and sampling parameters.
4. Score outputs blind (without knowing which model produced which output) to avoid confirmation bias.
5. Note not just accuracy but also latency, consistency, and formatting compliance.

This takes an afternoon and tells you more than any published leaderboard. Model rankings on general benchmarks shift as new models are released; the benchmark that matters is the one that matches your use case.

Matching Model Families to Task Types

Task type	Strong open-weight options
General instruction following, Q&A	Qwen3-8B, Llama 3.1 8B, Gemma 3 12B
Long-document summarization	Qwen3-14B/32B, Llama 3.1 70B (128K context)
Multilingual	Qwen3 (100+ languages), Gemma 3 (140+ languages)
Coding and software agents	Devstral Small 1.1, Qwen3-Coder, DeepSeek-V3
Math and reasoning	Qwen3 with thinking mode, DeepSeek-R1 variants
High-quality batch processing	Llama 3.1 405B, Qwen3-235B-A22B

The Real-Time Data Problem

All open-weight models share one structural limitation: their knowledge is frozen at training time. A Qwen3 or Llama 3.1 model does not know what happened last week. For workloads that require current information, whether that is current documentation, recent news, live pricing, or any time-sensitive data, local inference alone is insufficient.

The standard pattern is to retrieve relevant content at query time and include it in the prompt. For internal documents, a vector database or keyword search engine handles retrieval. For web data, you need a search API with real-time access.

You.com's Web Search API returns LLM-ready results from the live web via a simple API call. The free MCP endpoint at https://api.you.com/mcp?profile=free requires no signup and allows 100 queries per day, suitable for development. The full platform at you.com/platform adds you-answer (grounded synthesis), you-contents (clean page content), and you-research (multi-step cited synthesis). New accounts get $100 in complimentary credits. The Python SDK is youdotcom; docs are at you.com/docs. (You.com docs, 2026-09-04)

Local inference for privacy and latency. External search for freshness. The combination is where practical AI applications live.

Quantization and Its Effect on Model Tier

Quantization can change which tier a model fits in. A 14B model at Q4_K_M uses roughly 8-9 GB and fits in Tier 1. The same model at Q8_0 uses roughly 14 GB and fits only in Tier 2. Choosing a lower quantization level to fit a larger model on smaller hardware is a legitimate tradeoff, but it needs to be validated. The quality difference between Q4_K_M and Q8_0 is small for most tasks; the quality difference between Q2_K and Q8_0 is significant and will affect reasoning-heavy tasks noticeably.

GGUF is the standard format for quantized models used by llama.cpp and Ollama. vLLM supports GPTQ, AWQ, FP8, and GGUF. Hugging Face hosts GGUF files for most major models under ggml-org organization or as community uploads. You can pull GGUF directly in llama.cpp:

llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF

When evaluating quantization levels, the right approach is to test on a sample of your real inputs. Do not assume that a Q4 model is "good enough" without checking on your specific task. Domain-specific or knowledge-intensive tasks may show more quality degradation at lower quantization than general instruction following does.

Context Length and Its Hidden Costs

Context length appears in model specifications as a maximum (e.g., "128K context"). What is less often stated: filling that context costs VRAM. The KV cache that holds the context grows linearly with context length and batch size. A 7B model with 128K of actual context in use may require 8-16 GB of additional VRAM beyond the model weights themselves, depending on the serving implementation and precision.

vLLM's PagedAttention mitigates this by allocating KV cache in pages rather than contiguous blocks, reducing fragmentation and allowing more efficient use of available VRAM. Ollama's llama.cpp backend has a default context length that you can override at startup:

OLLAMA_CONTEXT_LENGTH=32768 ollama run qwen3:8b

For most workloads, setting a context length matched to your actual input distribution is more important than maximizing it. A 128K context limit is useful when you actually have 128K of relevant input; otherwise it is wasted VRAM reservation.

Practical Setup Recommendation

If you are starting fresh today with a single developer GPU:

• 8 GB VRAM: Start with Qwen3:8b via Ollama. It fits, it is capable, and it is easy to swap.
• 24 GB VRAM: Try Qwen3:32b for general use or Devstral if your primary use is coding.
• 32+ GB Apple Silicon: The Ollama MLX backend now delivers strong performance. Try Gemma 4 or Qwen3-32B.
• Multi-GPU server: Deploy vLLM for production serving. Consider DeepSeek-V3 or Llama 3.1 70B depending on task complexity.

In every case: evaluate on your real task before making the model your permanent choice. The landscape shifts every few months, and the winner in a benchmark announcement may not be the winner for your workload.