Self-hosted AI means an artificial intelligence model running on a server you control: your documents stay in, and nobody can change the tool on your behalf. For a small or mid-sized business, the right answer is neither "everything in-house" nor "everything in the cloud", but a clear split: confidential work stays with you, and what can leave goes to a provider.
Key takeaways
- With an online AI service you rent a tool: the provider chooses the model, the price, the terms and the retirement date.
- The three big providers retire models regularly; Anthropic commits to a minimum of 60 days' notice.
- A US provider remains subject to the CLOUD Act, even when your data is stored in Europe or Switzerland.
- Closed models keep a measurable lead: an external API is still the right choice for hard, non-sensitive tasks.
- The useful decision comes down to one question: which data goes where?
Why self-hosted AI is back on the management agenda
Three reasons, none of them technical.
Confidentiality. When an employee pastes a contract or a client file into an online tool, that data goes to a processor, often abroad. In Switzerland, the Data Protection Act governs disclosure abroad (art. 16 FADP); I explain what that means in my articles on the Swiss Data Protection Act and AI and on shadow AI.
Sovereignty. Since 2018, the US CLOUD Act has required a provider subject to US law to preserve or disclose the data it holds, whether that data is located in the United States or elsewhere. On the other side, the GDPR states that a decision by a third-country authority requiring a transfer of personal data can only be recognised if it is based on an international agreement (art. 48). The provider is caught between two legal systems, and your data with it; a server you control takes you out of that equation.
Dependence. The least visible one. An online AI model has an end date. On 14 April 2026, Anthropic announced that two of its models would be retired on 15 June 2026. On 11 June 2026, OpenAI announced that its GPT-5 family would shut down on 11 December 2026. Google announced the shutdown of Gemini 2.0 Flash for 1 June 2026 at the earliest. After that date, requests fail, and your automations have to be retested on another model. Prices move too, in both directions, as I showed in my article on LLM API costs. An open model installed on your premises only changes on the day you decide. For the rest of the cloud, the EU Data Act removes switching charges from 2027: see this article.
When an external API is still the right choice
Stanford University's 2025 AI Index report found that within one year, the performance gap between the best open and closed models had shrunk from 8% to 1.7% on some benchmarks. The 2026 edition adds a caveat: the gap reopened in 2025. In March 2026, the top closed model led the top open model by 3.3% on the Arena leaderboard, up from 0.5% in August 2024, and six of the top ten models were closed.
For long reasoning or complex code, the large online models are still ahead. For summarising, sorting emails or searching your procedures, a well-chosen open model is often enough. I compare the models in my comparison, and the hardware you need in this article.
Hence the hybrid approach, in three levels:
- Public or internal (website copy, routine emails): external API under contract, no second thoughts.
- Confidential (quotes, figures, ordinary personal data): API under a verified contract or local model, depending on your sector.
- Secret (files covered by professional secrecy, health, wealth): local model, and nothing leaves.
Under the hood: for your IT person
Serious self-hosted AI is a small stack of open-source building blocks, each one replaceable. Licences checked on the official repositories on 15 November 2026.
- Inference engine. Ollama (MIT) to get started quickly; llama.cpp (MIT) for quantised models in GGUF format, including without a GPU; vLLM (Apache 2.0) when several users query the model at the same time. All three expose an OpenAI-compatible API: your tools plug in without rewriting.
- Interface. Open WebUI connects to Ollama and to any OpenAI-compatible API, local or external. Watch out: its licence is no longer a standard open-source licence. It is a 3-clause BSD plus a clause that forbids removing the "Open WebUI" branding beyond 50 users over 30 days, unless you hold an enterprise licence.
- Searching your documents (RAG). pgvector (PostgreSQL licence) adds vector search to PostgreSQL: your indexed documents live in a database you already know how to back up and secure.
- Gateway. LiteLLM (MIT, except its "enterprise" directory) puts a single OpenAI-compatible API in front of local and external models. That is where the "which data goes where" rule is enforced, with keys, quotas and a budget per team.
- Logging. Record who sent what to which model. These logs contain personal data themselves: set a retention period and restrict access.
- Backup. Model weights can be downloaded again; the vector database, conversations and configuration cannot. Test the restore.
- Pitfalls. Pin the exact model version rather than a "latest" tag, or you recreate the retirement problem in-house. Never expose the inference engine directly to the internet: put it behind a VPN or authentication.
What I have learned
I have applied this split since 2025. In my home automation infrastructure, speech recognition and speech synthesis run locally, offline, while reasoning goes through an external API. My agents' memory lives in a database I host; non-sensitive tasks go to a large online model, for about 46 dollars of API usage a month.
I have also lived through dependence from the inside, twice. For my ERP, the vendor started charging for my customisations at renewal; I ended up migrating Odoo to my own server. For my agents, a third-party gateway reused my subscription in a way the provider's terms restricted more and more strictly; I rebuilt my setup on the provider's official tools, as I describe in the claudeVPS article.
My takeaway: what matters is not hosting the whole model, but keeping your data, your memory and the ability to change provider without starting over.
Where to start
1. Sort your data into three levels
One page, with examples from your business: what can leave, what needs a verified contract, what stays with you.
2. Pick a first local use case, modest and measurable
For example, searching your internal procedures or summarising files. Test an open model on your own documents before buying hardware.
3. Set up a single gateway
Every use goes through the same entry point, which sends each request to the local model or the external API according to the rule. If a provider retires a model, you change a setting, not your tools.
Frequently asked questions
Do I need to buy a server with a graphics card?
Not to start with: a small quantised model runs on a recent processor, more slowly. Sizing depends on the number of users; I cover it in the hardware article.
Is a local model good enough for my business?
For everyday tasks on your documents, often yes; for the hardest reasoning, no. Test it on a real case from your business.
Is self-hosted AI cheaper?
Not necessarily. You replace a usage-based bill with hardware, electricity and maintenance. What you mainly buy is control over your data, your versions and your schedule.
Going further
I set up infrastructure that stays with you: a local model when confidentiality requires it, an external API when that is acceptable, and a gateway that enforces the rule. The guides for my personal AI server detail the technical stack. To discuss your case: a 20-minute discovery call.
Build your own AI server
My guide 0, free, presents the architecture, the choice of host, sizing and costs. The Guides Pack (CHF 149) then walks through a complete stack, documented and tested in production: secure server, backups, secrets, memory, agents, API cost control and monitoring. If you would rather have support, the assisted installation (CHF 597) installs the foundation with you, in one or two video sessions.
Sources
- AI Index Report 2026, Technical Performance chapter, Stanford HAI, 2026 (accessed 15 November 2026).
- The 2025 AI Index Report, Stanford HAI, 2025 (accessed 15 November 2026).
- Model deprecations, Anthropic (accessed 15 November 2026).
- Deprecations, OpenAI (accessed 15 November 2026).
- Gemini API deprecations, Google (accessed 15 November 2026).
- 18 U.S. Code § 2713 (CLOUD Act, Pub. L. 115-141), Legal Information Institute, Cornell Law School, enacted 23 March 2018.
- Regulation (EU) 2016/679 (GDPR), art. 48, EUR-Lex, 27 April 2016.
- Federal Act on Data Protection (FADP), SR 235.1, art. 16, Swiss Confederation, status as of 1 September 2023.
- Official repositories and licences: Ollama, llama.cpp, vLLM, Open WebUI, pgvector, LiteLLM, GitHub (accessed 15 November 2026).
Cover photo: Derrick Coetzee / Wikimedia Commons, public domain (CC0).
Tags: self-hosted AI, local AI, digital sovereignty, data protection, open source, LLM