Self-Hosted LLM Serving: Picking a Stack That Survives Real Traffic

TLDR: A self hosted LLM is a model running on infrastructure you control, and the decision that decides whether it works is the serving stack. Ollama gets one developer productive in minutes, llama.cpp squeezes the most out of sparse hardware, and vLLM handles concurrent users. This guide matches each stack to a workload, shows a working deployment of each, covers the retrieval layer a self-hosted model needs for current information, and gives you the two tests that catch silent failures before your users do.

What is a self hosted LLM? It is a large language model whose weights and inference both run on hardware you control: a workstation, a rack in your data center, or a private cloud instance. Requests never leave your network boundary, and you decide when to upgrade the model, at the cost of buying and maintaining the machinery.

What Does Self-Hosting an LLM Require Beyond Downloading Weights?

Self-hosting has three layers, and most failed projects treat only the first as the project. The model layer is the weights: open-weights families like Llama 3.1 and Qwen3 are downloadable, and their licenses permit self-hosting in commercial products with conditions you should read before shipping. The serving layer is the software that loads weights onto a GPU and answers HTTP requests, and it is where this guide lives, because the right serving stack depends entirely on your concurrency and hardware. The retrieval layer is the part teams forget: model weights are frozen at training time, so anything that changed since, from prices to library APIs to news, requires a search call at request time with results passed in as context.

A fourth layer hides under the other three: operations. Drivers, CUDA versions, model refresh cadence, and request logs all need an owner. The teams that succeed at self-hosting assign that owner explicitly.

Which Serving Stack Fits Your Workload?

Match the stack to the workload before installing anything, because each one optimizes for a different failure. The decision axis is concurrent users: one, a handful, or a production service.

• Ollama is the right start for a single developer or small team. It bundles model download, quantization management, and an HTTP API on localhost:11434, and it swaps models with one command. Its scheduler is built for interactive single-user use, so request queuing degrades under sustained concurrency.
• llama.cpp server is the choice for constrained or unusual hardware: CPU-only boxes, older GPUs, machines where you want minimal dependencies. It is the engine under Ollama anyway, exposed directly, and its memory mapping lets a machine serve a model larger than its VRAM by spilling layers to CPU. Expect token throughput to drop accordingly.
• vLLM is the production choice for concurrent users. It implements paged attention, which treats the key-value cache like virtual memory pages instead of pre-allocated blocks, so memory that would sit wasted in padding gets used for more concurrent sequences. Continuous batching batches new requests into running ones instead of waiting for a batch to finish.

The trade-off, named: convenience against throughput. Ollama optimizes time-to-first-token for one person, vLLM optimizes aggregate throughput for many, and llama.cpp optimizes hardware range. A stack that is wrong for the workload fails quietly, as latency creep rather than an error.

How Do You Deploy Each One?

Each stack has a documented minimal deployment. These commands are the verified starting points from each project's own documentation.

Ollama, per its install docs: run the install script, pull a model, and the API is live:

curl -fsSL https://ollama.com/install.sh | sh
ollama pull llama3.1:8b
curl http://localhost:11434/api/chat -d '{
  "model": "llama3.1:8b",
  "messages": [{"role": "user", "content": "ping"}]
}'

llama.cpp, per its server build docs, compiles with CUDA support and serves an OpenAI-compatible endpoint:

cmake -B build -DGGML_CUDA=ON && cmake --build build --config Release
./build/bin/llama-server \
  --model llama-3.1-8b-instruct.Q4_K_M.gguf \
  --host 0.0.0.0 --port 8080 \
  --n-gpu-layers 99

vLLM, per its quickstart, installs with pip and serves an OpenAI-compatible API with continuous batching on:

pip install vllm
vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.90

One planning note that bites people: context length is a memory budget. Each concurrent sequence holds its own key-value cache for the whole context window, so eight concurrent requests at a 128K window can exhaust a GPU that runs the same model happily for one request. Cap --max-model-len at what your product actually needs, not what the model supports.

How Does a Self-Hosted Model Get Current Information?

It cannot, by itself. The standard pattern is retrieval at request time: your application calls a search API, passes the results to the local model as context, and the model answers from that context. The model and prompts stay on your hardware, and only the query string and public web results cross the boundary.

The You.com Web Search API is built for exactly this role. One request returns web and news results with snippets, source URLs, and metadata, and it supports freshness filters, country and language parameters, and domain restrictions. Wiring it into a local serving stack is a few lines:

import json, urllib.request

def search(query: str, key: str) -> list:
    req = urllib.request.Request(
        "https://api.you.com/v1/search",
        data=json.dumps({"query": query, "count": 5}).encode(),
        headers={"Content-Type": "application/json", "X-API-KEY": key},
    )
    with urllib.request.urlopen(req, timeout=30) as resp:
        return json.load(resp).get("results", {}).get("web", [])

context = "\n".join(
    f"{r['url']}\n{(r.get('snippets') or [''])[0]}"
    for r in search("vllm paged attention tuning guide", KEY)
)
# pass `context` to your local model alongside the user question

For extracting the full text of specific URLs you already hold, the Contents API returns markdown or HTML on demand, which fits pipeline steps that need whole documents rather than snippets.

What Failure Modes Should You Instrument From Day One?

Two failures are silent, which is what makes them expensive.

Context truncation. Serving stacks apply their own working context limits, and when a request exceeds one, nothing errors. The prompt gets cut, usually from the middle, and the model confidently answers a different question. Detection is a canary: place a unique token early in a long prompt, ask the model to repeat it, and alert when the repetition fails. Run this check after every config change.

Latency creep under concurrency. An undersized stack degrades as queue depth grows, and users experience it as the assistant "getting slower this week" rather than as an outage. Instrument per-request queue wait separately from inference time, and load-test with your real concurrency number before calling the deployment production-ready. A stack that serves one user in 300 milliseconds can take multiple seconds per user at eight concurrent requests, and that difference does not show up in single-user smoke tests.

How Do You Keep the Stack From Decaying?

Self-hosted inference is a living system. Three habits keep it healthy: pin your model versions in configuration rather than tracking "latest", so an upgrade is a deliberate change with a regression suite behind it. Keep a set of about twenty real production prompts with known-good answers and run them after every model, driver, or serving-stack update. And watch upstream security notices for the model families you serve, because weight updates and tokenizer changes occasionally arrive together and break prompt caching or tool parsing in ways the regression suite is your only defense against.

Where Does Self-Hosting Stop Being the Right Answer?

Be honest about utilization. GPU hardware amortizes, so sustained high-volume inference can cost less per request than per-request API billing, but a GPU that idles most of the day is a bad lease. If your traffic is bursty, unpredictable, or still small, a hosted API with a contractual zero data retention agreement may satisfy the same privacy requirement at a fraction of the operations burden. The full build-versus-buy walkthrough, including the boundary test for compliance-driven deployments, is in the on-premise AI guide. For model selection by size, context window, and license, see the local AI models shortlist. The step-by-step Ollama walkthrough, including the truncation canary, is in how to run an LLM locally. And the hub page for the whole cluster is Local LLM: Running Large Language Models on Your Own Infrastructure.

The next action, if you are starting today: pick the stack that matches your concurrency, deploy it with the context length capped to your real need, wire the truncation canary before the first real user, and load-test at your target concurrency before you announce anything.