NL · EN · DE

Why I run my AI locally (and when I don't)

Most of the AI traffic in my house now runs through my own hardware. No API keys, no per-token invoices, and no text leaving my network until I decide it should. This is what runs, why, and — honestly — where it stops.

The setup

Two machines do the work:

  • Orac — an MSI B450 board with an RTX 3090 (24 GB) and 128 GB of RAM. It serves language models through a self-hosted inference service (Strata), and speech synthesis with XTTS-v2: cloned voices and audiobooks in Dutch, Frisian, English and Polish.
  • Roggy — a NUC-class machine with a TITAN RTX (24 GB) for training tasks and heavier TTS runs.

Plus a mini-NUC with an Intel Arc iGPU for OpenVINO experiments: proof that "local AI" doesn't require a big GPU, as long as the model is small enough.

Where local AI wins

Privacy isn't an argument any more, it's the default. Calendar, mail, screenshots of receipts — none of it goes anywhere. For a website about computing that sounds obvious; in practice it's the one reason that survives every question.

Costs are flattened. A few-hundred-euro 3090 runs for years. Anyone pushing tens of thousands of lines of code or summaries through a cloud model daily pays that back every month; here it just shows up on the electricity bill.

Speed where it counts. A small local model answers "summarise this log file" faster than a cloud request with network + queue time. For agent loops, where one task makes dozens of small model calls, that adds up enormously.

Where it stops

Locally I run what fits in 24 GB and what I can judge myself: summarising, translating, coding chores, speech. For the genuinely hard reasoning task — architecture decisions, a nasty debugging session with lots of context — my agent reaches for a cloud model. The line falls exactly where "cheap and frequent" turns into "rare and critical".

To be fair: local models need maintenance. Drivers, quarantine-testing after an update, a machine that crashes and needs to come back up (see the WOL scripts I wrote for that). If you only ask a question now and then, the cloud is the better deal.

The thread

For me AI isn't an oracle but a tool: something you put in your own cupboard, can audit, and can unplug at any moment. In the next posts: how an agent (Hermes) manages my servers with it, and how local talking and listening works with XTTS and Piper.

← Back to the blog overview