This article is me ruminating over a problem I have as a local AI app developer. And I admit that I am having a very Philip K. Dick “Do Androids Dream of Electric Sheep?” moment as I start this post. Why? Because this post is about AIs sleeping, specifically an AI project I am working on called Companion.
The Problem… my problem
The problem that I have is I own a PC with only 16GB of VRAM, yet I want to create local LLM companions for chatting, coding, etc. Now, if you’re bothering to read this, then you know that building such local-first AI companions introduces challenges. Over time:
- Conversations become too large to fit into the model’s context window.
- User facts accumulate and may become outdated or contradictory.
- Session histories grow indefinitely.
- Information needs to survive restarts without relying on cloud services.
Left unchecked, an AI companion gradually becomes slower, less efficient, and more prone to recalling outdated information. My solution was borrowed from biology: a dedicated sleep cycle that actively manages memory rather than simply storing everything forever. Naturally, this is not a unique idea. And though I didn’t know it at the time, Google had been working on something similar (Language Models Need Sleep: Learning to Self-Modify and Consolidate Memories). However, the difference, as it turns out, was that I was looking at the solution at the memory architecture layer rather than the neural training layer.
This said, I do have some assets sitting around. Like any “old-school” IT bloke, I have old PCs floating around. In fact, I have them networked and call them my home lab. I wondered how I might use this. The obvious answer was a distributed system using the resources of my network. So I designed a distributed deployment, meaning Companion may run across multiple machines. For example, in my lab:
- My primary PC handles user interaction and voice processing.
- My weakest PC hosts a FastAPI service that proxies requests to Ollama.
This separation allows resource-intensive workloads to be distributed across available hardware.
What Is “Sleep” in Companion?
But back to sleep. Sleep is a structured maintenance process that runs during periods of inactivity or when explicitly triggered by the user. Instead of suspending execution, the system moves through several stages:
Each stage helps transform raw conversation history into useful long-term knowledge.
The Architecture Behind Sleep
Database Layer
The database serves as the source of truth for:
- Sessions
- Conversation turns
- Memories
- User facts
Using SQLite provides persistence across restarts while keeping the system entirely local.
The Five Stages of Sleep
1. Idle Detection
Sleep begins when Companion detects either:
- A period of inactivity, or
- An explicit user command
At this point, the system determines that it’s safe to perform maintenance activities.
2. Consolidation
Fact Consolidation
Facts extracted from recent conversations are processed using a lightweight model such as qwen2.5:1.5b. The system then:
- Normalizes text
- Removes duplicates
- Merges related facts
- Assigns importance scores
- Applies recency weighting
Example: “likes coffee” + “drinks coffee daily” becomes “coffee enthusiast”or another normalized representation chosen by the consolidation process.
Memory Consolidation
Memories are re-evaluated using the retrieval scoring model. During this stage:
- High-value memories are retained.
- Medium-value memories may be compressed.
- Low-value memories may be archived or removed.
This prevents memory stores from becoming cluttered with information that is unlikely to matter in future conversations.
3. Pruning
Pruning keeps the system healthy. Without pruning, memory systems tend to accumulate noise. Hence, “forgetting is a feature, not a bug”. Biological memory actively forgets for a reason, and this is central to why this system stays fast and relevant over time. Examples of information that may be removed include:
- Contradicted facts
- Obsolete preferences
- Repeated statements
- Low-importance historical details
This ensures that retrieval remains both fast and relevant.
4. Persistence
After consolidation and pruning are complete, the updated state is committed to disk. This includes:
- Refined memory structures
- Consolidated user facts
- Updated indexes
- Session summaries
In distributed environments, synchronisation must also occur between participating nodes.
5. Rest State
Once processing finishes, Companion enters a lightweight resting state. At this point:
- Resource utilisation drops.
- Databases are consistent.
- Memory structures are optimised.
- The system is ready to wake instantly when needed.
Context Compression: Solving the Context Window Problem
One of the biggest challenges in long-running AI conversations is limited context size. Even models with large context windows eventually reach their limits (“context rot”). Companion addresses this through context compression. Older conversation turns can be:
- Summarized
- Condensed
- Archived
while preserving important information within the system’s structured memory and fact stores. The result is that Companion remembers what matters without needing every historical message available at all times. In all honesty, this reminds me of how I think because I can’t remember the details of some conversations I have with people, but I remember the gist of them.
All this said, I’m still at the beginning of my journey with this project. One obvious issue that I need to address, given it’s a distributed system, is that if both nodes attempt consolidation independently, they may arrive at different memory states. This creates a classic distributed systems problem known as a split-brain scenario, where two systems believe they are authoritative. So my future implementations must coordinate sleep activities across nodes to ensure a single consistent memory state.
So how does this compare to Google’s sleep?
The core philosophical idea is almost identical:
Intelligence should not simply accumulate experiences. It should periodically reorganize them.
In my Companion design, sleep is when:
- facts are deduplicated
- memories are re-ranked
- stale information is removed
- conversation history is compressed
- state is persisted across sessions
In Google’s framework, sleep is when:
- temporary knowledge is consolidated
- short-term memory is transferred into more persistent representations
- rehearsal and self-improvement occur
- forgetting is mitigated through offline consolidation
Both systems draw direct inspiration from biological memory consolidation during human sleep.
Where I feel my design Is actually more practical for the home enthusiast like me is Google’s work focuses on:
- modifying model parameters
- continual learning
- avoiding catastrophic forgetting
- consolidating knowledge into neural weights via training processes
My approach focuses on:
- fact stores
- SQLite persistence
- retrieval systems
- vector memories
- long-term user relationships
- distributed synchronization
This means that my solution works on someone’s current hardware and doesn’t require a specialised continual-learning infrastructure.
Final Thoughts
The most interesting aspect of Companion’s sleep system is that it treats memory as an active process rather than passive storage. Instead of endlessly accumulating data, the system continuously refines what it knows:
- Consolidating useful information
- Eliminating noise
- Compressing history
- Preserving important memories
In many ways, this mirrors how biological memory works. The result is an AI companion that becomes more organised, more efficient, and ultimately more useful over time.
Sleep, in my approach, isn’t downtime. It’s when the system transforms experience into knowledge.
