How to Run LLMs Locally: A Complete Guide to Ollama, llama.cpp, and vLLM

Running large language models locally gives you full control over data, predictable costs, and offline access. With tools like Ollama, llama.cpp, and vLLM, you can run models ranging from small 1B parameter models to 70B+ models depending on your hardware. The right choice depends on whether you prioritize simplicity, customizability, or high-throughput serving.

Why Run LLMs Locally?

Local inference eliminates API latency variability and removes the need to send sensitive prompts to third-party providers. For many workflows, it also caps costs at hardware and electricity. Key benefits include privacy (data never leaves your machine), offline capability, reproducibility, and freedom to experiment with open models without rate limits.

Tradeoffs include upfront hardware costs, higher power consumption, and more operational responsibility. Frontier proprietary models may still outperform smaller local models on niche tasks, so the decision is often task-specific rather than absolute.

Hardware Requirements

Memory is the limiting factor for most local LLMs. CPU inference is possible for small models, but GPU acceleration dramatically improves tokens per second. The table below provides rough guidance for 4-bit GGUF quantization (Q4) as a common baseline.

Note: Higher precision (FP16, GGUF GGUF-v2 without heavy quantization) requires more memory. Aggressive quantization (GGUF IQ2, GGUF Q2) saves memory but can reduce quality on complex tasks.

Quantization Basics

Quantization reduces model weight precision to shrink size and speed up inference. Two common formats dominate local inference:

• GGUF (llama.cpp family): A portable, mmap-friendly format optimized for CPU+GPU inference on consumer hardware. Common quantizations include GGUF Q2_K, Q3_K_M, Q4_K_M, GGUF Q5, and GGUF GGUF Q6.
• Safetensors/HF (vLLM): HuggingFace safetensors weights in bfloat16/float16 or GPTQ/AWQ quantized variants. vLLM loads these directly for paged attention and high-throughput serving.

For most general tasks, GGUF Q4_K_M strikes a strong balance between quality, speed, and size. If you need maximum quality on a capable GPU, bfloat16 weights with vLLM are often preferred.

Tool Comparison

1. Ollama: Fastest Path to Running Locally

Ollama is a CLI+daemon that abstracts model download, quantization selection, and API exposure. It's ideal for getting started with minimal configuration.

# Install Ollama (Linux/macOS/Windows)
curl -fsSL https://ollama.com/install.sh | sh

# Pull a model
ollama pull llama3.1:8b

# Run interactively
ollama run llama3.1:8b

# List models
ollama list

After starting, you can use http://localhost:11434/v1 with OpenAI-compatible clients, or pair with Open WebUI for a ChatGPT-like interface. Ollama also supports custom Modelfile definitions to fine-tune system prompts, parameters, and adapters.

2. llama.cpp: Maximum Portability and Control

llama.cpp is a C/C++ implementation of transformer inference optimized for CPU and GPU across platforms (x86_64, ARM, Metal, Vulkan). It's the foundation for many local tools and supports GGUF files from HuggingFace and other sources.

# Clone and build llama.cpp
git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build --config Release

# Download a GGUF model (example)
# Run server with OpenAI-compatible API
./build/bin/llama-server -m ./models/mistral-7b-instruct-v0.3.Q4_K_M.gguf --port 8080

llama-server exposes an OpenAI-compatible API, while llama-cli is great for quick interactive testing. Advanced users can tune context size, threads, batch size, and GPU offload layers for specific hardware.

3. vLLM: High-Throughput GPU Serving

vLLM uses paged attention to maximize GPU memory utilization and handle many concurrent requests efficiently. It's designed for serving OpenAI-compatible endpoints at scale on NVIDIA GPUs, making it well-suited for production workloads and agent backends.

# Install vLLM (requires CUDA for GPU)
pip install vllm

# Start OpenAI-compatible server
vllm serve mistralai/Mistral-7B-Instruct-v0.3   --dtype auto   --tensor-parallel-size 1   --max-model-len 4096

Key features include continuous batching, prefix caching, speculative decoding, and multi-GPU sharding. If you need to serve many users or run long-running agent tasks with high concurrency, vLLM typically delivers the best tokens-per-second-per-request under load.

Frontends and UX

Local models are most useful with a clean interface. Popular options include:

• Open WebUI: Feature-rich ChatGPT-like UI with RAG, function calling, user management, and multi-model support.
• LM Studio: Cross-platform GUI for downloading GGUF models, tweaking parameters, and chatting locally.
• AnythingLLM: Local workspace combining models, documents, and knowledge bases with a polished UI.

All work with OpenAI-compatible endpoints, so you can switch backends (Ollama, llama.cpp server, vLLM) without changing the UI.

Performance Expectations

Real-world performance varies by hardware, quantization, context length, and batch size. As rough guidance for 7–8B models at typical context (2K–4K):

• CPU-only (modern x86): 5–15 tokens/sec (Q4)
• Apple Silicon (M-series): 40–80 tokens/sec (Metal)
• Mid-range GPU (8–12GB VRAM): 40–90 tokens/sec
• High-end GPU (16GB+ VRAM): 80–150+ tokens/sec

Larger context windows, longer outputs, and concurrent requests will reduce these numbers. Techniques like prefix caching (vLLM) and context shift optimization help in real workloads.

When to Choose Local vs API