Skip to Content

AI assistant memory: make it remember, on your server

Keep the essentials, find by meaning, link memories together: a persistent memory hosted on your own server.
November 15, 2026 by
AI assistant memory: make it remember, on your server
Federico Solutions, Myrko Federico

AI assistant memory: make it remember, on your server

Field: AI assistants, persistent memory, self-hosting

For: SME owners who work with an AI assistant and are tired of explaining everything again, and their IT person

Experience: the memory I added to my AI assistants, hosted on my server and used every day

An AI assistant answers well, but the next day it has forgotten everything: your projects, your decisions, your preferences. You brief it every morning like a newly arrived intern. This is not a flaw in your tool, it is how a language model normally works. The solution is to give it a memory: a place where it stores what it learns, and where it looks for what is useful before answering. Here is how I built it, what it keeps, and why I keep it on my server.

Background

In an article on the limits of LLMs without memory, I explained why a model forgets: it only sees what fits in its context window, for the length of a conversation. And in Fine-tuning or RAG, I compared the ways of giving knowledge to an AI. An assistant's memory is a special, very concrete case: the point is not to teach it your catalogue, but to make it remember your work in progress, from one day to the next.

The problem: an assistant that starts from scratch

  • Context to repeat: every conversation starts with the same explanations about the company, the customers, the tools.
  • Lost decisions: what was settled last week no longer exists for the assistant, which suggests the rejected option again.
  • The provider's memory: consumer assistants offer a memory, but it is stored on their side, and you choose neither what it keeps nor where.
  • Notes in a heap: keeping everything in one long file ends up costing a lot, because the assistant rereads it all every time, and becomes unreadable.

The solution: a memory in five moves

1. Keep the essentials, not everything

At the end of a conversation, the assistant summarises what is worth keeping: a decision and its reason, the status of a project, a preference, a pitfall encountered. A useful memory looks like a good project manager's notebook, not a full recording of every meeting.

2. Find by meaning

Each memory is turned into a vector, a kind of fingerprint of its meaning, and stored in a vector database. When you ask "what did we decide for this customer?", the assistant finds the right memory even if the words are different. It is the RAG principle, applied to your own history rather than to your documents.

3. Link memories together

A knowledge graph links people, projects, decisions and tools. It makes it possible to answer questions that span several memories: which projects depend on this supplier, who took a given decision, what has changed since.

4. Recall automatically before answering

Before each answer, the most relevant memories are added to the assistant's context, in limited number. It does not need to think of searching: it remembers, like a colleague who knows the file.

5. Separate what is shared from what is personal

A common memory for the company, one memory per person, possibly one per agent. What one colleague tells the assistant must not resurface in the answer given to another.

Why keep it in-house

An assistant's memory quickly becomes the most sensitive document in the company: it summarises your customers, your prices, your decisions and your difficulties. Hosted on your server, it stays under your control: you know where it is and who reads it, and you can back it up, correct it and delete what needs to be deleted. And it survives a change of tool: if you switch assistants tomorrow, your memory is still there.

What I learned running it

  • Memory is the layer that changes everything. The day I no longer had to explain my projects again every morning, the assistant became a work tool, and not just a question-and-answer one.
  • Too compartmentalised, it forgets in another way. I had separated memory by project. A lesson learned on one stayed invisible to the others, and the same mistake happened again elsewhere. I added a shared log of pitfalls, checked before any work that touches several projects.
  • One memory at a time. Two memory systems active at the same time step on each other: you no longer know which one writes and which one answers. I disabled the tool's built-in memory when I plugged in mine.
  • It gets updated like everything else. Updating one of the components without the other has already made two parts of my memory drift apart. I update them together, by hand, after a backup.
  • Local does not mean offline. The meaning fingerprints are computed on my server, but summarising and extracting links goes through a language model, an external one in my setup. You need to know this to decide what you entrust to the memory, or use a local model.

Under the hood (for your IT person)

  • Engine: Cognee (Apache 2.0), which combines vector search and a knowledge graph, in a container on an internal network, API reachable only locally.
  • Storage: PostgreSQL with the pgvector extension for vectors, complemented by Redis.
  • Embeddings: computed locally with FastEmbed (ONNX, no GPU); no cost and nothing sent outside for this step.
  • Integration: automatic recall before each turn (number of memories, minimum score and token budget capped), end-of-session summary, shared, user and agent scopes; an MCP server to query the same memory from other tools.
  • Operations: image built locally and pinned, manual update with a prior snapshot and a planned rollback, database backed up with the rest of the server.

What I take away

An assistant without memory is an excellent consultant who never comes back. With a well-designed memory, selective, searched by meaning and kept on your premises, it becomes a colleague who knows your files. It comes into its own with a team chat to run the assistant from, and it is one of the building blocks of claudeVPS, where all my work sessions share the same memory.

Going further

I design AI assistants that know your business, with a memory hosted on your premises and clear rules about what it keeps. See custom applications and automations, or book a 20-minute discovery call.

The step-by-step guide

For your IT person, guide 5 in the Open Source Server Stack series, persistent memory (CHF 39), installs this memory in 2 to 3 hours: Cognee with pgvector and local embeddings, automatic recall before each answer, end-of-session summary, memory scopes, tests and an update procedure with rollback. At the end, you ask where a project stands, and the assistant answers with the decisions taken and the tasks in progress.

The Guides Pack (CHF 149) brings together guides 0 to 8 of the series: a complete stack, documented and tested in production, from the secure server to secrets, backups and monitoring. If you would rather have support, the assisted installation (CHF 597) installs the foundation with you in one or two video sessions, Complete Pack included, with 60 days of email support.

Sources

Tags: AI memory, AI assistant, persistent memory, vector database, knowledge graph, RAG, Cognee, self-hosting, SME

Self-hosted team chat to run your AI assistant
A messaging app you own, one account and one channel per purpose, alerts in the same place: your AI's control room.