Sovereign AI · Setup Guide · July 2026

Run your own LLM. Governed from day one.

What hardware you actually need in 2026 (with verified prices), how to serve an open-source model with vLLM or Ollama, and how to get back the spend caps, kill switch, and audit trail you lose when you leave hosted APIs. Total governance setup time: about 60 seconds.

Step 1

Pick your hardware tier.

TierModelsHardwareCost (2026, verified)
StarterMost SMB agent work: support triage, drafting, classification, tool calling 7B to 9BLlama 3.1 8B, Qwen 8B, Gemma class. Needs ~8GB VRAM quantized One 24 to 32GB GPU, or a Mac with 32GB+ unified memory RTX 5090 32GB: $1,999 MSRP but street prices ran ~$3,000 to $4,300 mid-2026 on the memory shortage. Used RTX 3090 24GB pairs are the budget path.
WorkhorseQuality close to hosted frontier for most business tasks 70B classLlama 70B, Qwen 72B. Needs ~35 to 48GB VRAM quantized: does NOT fit one consumer card Dual GPUs (2x RTX 3090 ~ $1,700 used), a Mac with 64GB+ unified memory, or one datacenter card ~$2,000 to $6,000 owned, or one rented H100 at ~$1.99 to $3.29/hr
Frontier openThe "own your intelligence, soup to nuts" play GLM large, DeepSeek R1/V3, Kimi K2 class MoE Multi-GPU server, typically 8x H100/H200 H100 purchase: ~$25,000 to $40,000 per card. Rented: multi-GPU nodes by the hour. Rent unless you run it 24/7.

Buy vs rent: rent first.

H100 rental prices fell roughly 64 to 75 percent from late 2024 to early 2026 (RunPod from $1.99/hr, Lambda from $2.49/hr, hyperscalers $2 to $11/hr, spot near $1.03/hr) while consumer card street prices went UP on GDDR7 shortages. Buying breaks even only at roughly 8 to 12 months of near-constant utilization. Renting GPUs inside your own VPC keeps weights and data in your perimeter with zero capex: that is the sovereignty most businesses actually need.

Step 2

Serve the model with an OpenAI-compatible endpoint.

Both mainstream servers expose the same API shape, which is what makes the governance layer drop-in.

Ollama (simplest, great for Starter tier)

$ ollama pull llama3.1:8b
$ ollama serve  # OpenAI-compatible API at http://localhost:11434/v1

vLLM (production serving, 2 to 4x more concurrent requests via PagedAttention)

$ pip install vllm
$ vllm serve meta-llama/Llama-3.1-8B-Instruct  # http://localhost:8000/v1
Step 3

Add the governance you just gave up.

Leaving a hosted API means losing its spend dashboard, rate limits, and usage logs. AgentGuard® restores them locally: it is a pure SDK, no gateway, no proxy, no vendor cloud in your traffic path.

$ npm install @agentguard-run/spend  # or: pip install agentguard-spend
$ agentguard init  # point it at http://localhost:8000/v1 or any OpenAI-compatible endpoint

Any endpoint the SDK does not recognize as a hosted provider is routed as self-hosted automatically: zero data retention, your jurisdiction, your keys. You get hard spend caps, capability gates, and a kill switch enforced in-process, before an action runs.

Step 4

Prove it with a signed receipt.

Every decision settles to an Ed25519-signed, hash-chained, content-free receipt recording which model ran, the origin of its weights (China-origin families like GLM, DeepSeek, Qwen, and Kimi are flagged automatically), the hosting jurisdiction, and the retention posture. Anyone can verify it independently at agentguard.run/verify, without trusting us or you. For a regulator, an auditor, or your board, that is the difference between "trust us" and "verify it yourself."

Start free (BYO key or BYO cluster) Why sovereign AI needs this →

Hardware prices verified against market trackers and provider price pages, July 2026. GPU pricing moves fast: treat ranges as directional and check current listings before purchasing. AgentGuard does not sell or resell hardware or cloud GPU capacity.