Why I run my AI locally (and when I don't)
Most of the AI traffic in my house now runs through my own hardware. No API keys, no per-token invoices, and no text leaving my network until I decide it should. This is what runs, why, and — honestly — where it stops.
The setup
Two machines do the work:
- Orac — an MSI B450 board with an RTX 3090 (24 GB) and 128 GB of RAM. It serves language models through a self-hosted inference service (Strata), and speech synthesis with XTTS-v2: cloned voices and audiobooks in Dutch, Frisian, English and Polish.
- Roggy — a NUC-class machine with a TITAN RTX (24 GB) for training tasks and heavier TTS runs.
Plus a mini-NUC with an Intel Arc iGPU for OpenVINO experiments: proof that "local AI" doesn't require a big GPU, as long as the model is small enough.
Where local AI wins
Privacy isn't an argument any more, it's the default. Calendar, mail, screenshots of receipts — none of it goes anywhere. For a website about computing that sounds obvious; in practice it's the one reason that survives every question.
Costs are flattened. A few-hundred-euro 3090 runs for years. Anyone pushing tens of thousands of lines of code or summaries through a cloud model daily pays that back every month; here it just shows up on the electricity bill.
Speed where it counts. A small local model answers "summarise this log file" faster than a cloud request with network + queue time. For agent loops, where one task makes dozens of small model calls, that adds up enormously.
Where it stops
Locally I run what fits in 24 GB and what I can judge myself: summarising, translating, coding chores, speech. For the genuinely hard reasoning task — architecture decisions, a nasty debugging session with lots of context — my agent reaches for a cloud model. The line falls exactly where "cheap and frequent" turns into "rare and critical".
To be fair: local models need maintenance. Drivers, quarantine-testing after an update, a machine that crashes and needs to come back up (see the WOL scripts I wrote for that). If you only ask a question now and then, the cloud is the better deal.
The thread
For me AI isn't an oracle but a tool: something you put in your own cupboard, can audit, and can unplug at any moment. In the next posts: how an agent (Hermes) manages my servers with it, and how local talking and listening works with XTTS and Piper.