Introduction to Agent Memory

Duration: 15 min

Overview

Memory is what transforms a stateless LLM into a capable AI agent. Without memory, every interaction starts from scratch - the agent cannot learn from past mistakes, recall user preferences, or build on previous conversations. This module introduces the core memory types used in modern agent systems and explains why memory architecture is the single most important design decision when building production agents.

You will learn the taxonomy of agent memory, understand the trade-offs between different approaches, and see how memory systems map to human cognitive models.

Key Concepts & Foundations

What: Agent memory is the mechanism by which an AI agent stores, organizes, and retrieves information across interactions

Why: Without memory, agents cannot maintain context, learn from experience, or personalize responses

How: Memory systems combine buffers, databases, and retrieval algorithms to give agents persistent state

The Memory Taxonomy

AI agent memory draws from cognitive science. The four primary types are:

1. Working Memory - The active context window; what the agent is currently thinking about 2. Short-Term Memory - Recent conversation history kept in buffers (minutes to hours) 3. Long-Term Memory - Persistent storage of facts, preferences, and knowledge (days to forever) 4. Episodic Memory - Records of specific past experiences and their outcomes

Why Memory Architecture Matters

A chatbot without memory forgets your name between messages

A coding agent without memory re-reads the same files repeatedly

A research agent without memory cannot synthesize findings across sessions

A personal assistant without memory cannot learn your preferences

Memory vs. Context Window

The context window is NOT memory - it is the agent's scratchpad. True memory involves:

Persistence - Survives beyond a single conversation

Selection - Retrieves only what is relevant, not everything

Organization - Structures information for efficient retrieval

Forgetting - Discards outdated or irrelevant information

Hands-On Implementation

from dataclasses import dataclass, field
from typing import Any
from datetime import datetime

@dataclass
class MemoryEntry:
    """A single memory record with metadata."""
    content: str
    timestamp: datetime = field(default_factory=datetime.now)
    memory_type: str = "general"
    importance: float = 0.5
    access_count: int = 0
    def access(self):
        """Track when this memory is retrieved."""
        self.access_count += 1
        return self.content

class AgentMemorySystem:
    """Basic agent memory with working, short-term, and long-term stores."""
    def __init__(self, working_memory_limit: int = 5):
        self.working_memory: list[str] = []
        self.short_term: list[MemoryEntry] = []
        self.long_term: list[MemoryEntry] = []
        self.working_memory_limit = working_memory_limit
    def add_to_working_memory(self, content: str):
        """Add to working memory, evicting oldest if at capacity."""
        self.working_memory.append(content)
        if len(self.working_memory) > self.working_memory_limit:
            evicted = self.working_memory.pop(0)
            # Move evicted items to short-term memory
            self.short_term.append(
                MemoryEntry(content=evicted, memory_type="short_term")
            )
    def promote_to_long_term(self, content: str, importance: float = 0.8):
        """Store important information in long-term memory."""
        entry = MemoryEntry(
            content=content,
            memory_type="long_term",
            importance=importance
        )
        self.long_term.append(entry)
    def recall(self, query: str) -> list[str]:
        """Simple keyword-based recall from all memory stores."""
        results = []
        query_words = set(query.lower().split())
        # Check working memory first (fastest)
        for item in self.working_memory:
            if query_words & set(item.lower().split()):
                results.append(f"[working] {item}")
        # Then short-term
        for entry in self.short_term:
            if query_words & set(entry.content.lower().split()):
                entry.access()
                results.append(f"[short-term] {entry.content}")
        # Then long-term
        for entry in self.long_term:
            if query_words & set(entry.content.lower().split()):
                entry.access()
                results.append(f"[long-term] {entry.content}")
        return results
    def get_context(self) -> str:
        """Build context string from working memory for LLM prompt."""
        return "\n".join(self.working_memory)

Demo: Agent that remembers user preferences
memory = AgentMemorySystem(working_memory_limit=3)
User interactions
memory.add_to_working_memory("User asked about Python decorators")
memory.add_to_working_memory("User prefers concise code examples")
memory.promote_to_long_term("User is an intermediate Python developer", importance=0.9)
memory.add_to_working_memory("User wants to learn about memory systems")
memory.add_to_working_memory("Current topic: agent memory architecture")
Now working memory only has the 3 most recent items
print("Working memory:", memory.working_memory)
print("Short-term (evicted):", [e.content for e in memory.short_term])
print("Long-term:", [e.content for e in memory.long_term])
Recall relevant memories
print("\nRecall 'Python':", memory.recall("Python"))

Advanced Techniques

When designing agent memory systems, consider these architectural principles:

1. Layered Retrieval - Check working memory before short-term before long-term (speed vs. completeness) 2. Importance Scoring - Not all memories are equal; score by relevance, recency, and frequency of access 3. Memory Decay - Implement time-based forgetting to prevent information overload 4. Consolidation - Periodically merge related short-term memories into summarized long-term entries 5. Memory Indexing - Use embeddings and vector search for semantic retrieval at scale

Quiz

Q1: What distinguishes true agent memory from a context window?

A) Memory is faster than context windows

B) Memory persists, selects, and organizes information beyond a single conversation ✓

C) Context windows are more expensive

D) Memory only stores text, context windows store everything

Q2: Which memory type holds the agent's current active thoughts?

A) Long-term memory

B) Episodic memory

C) Working memory ✓

D) Semantic memory

Q3: Why is memory architecture the most important agent design decision?

A) It determines the programming language used

B) It controls how the agent learns, personalizes, and maintains context across interactions ✓

C) It reduces API costs

D) It makes the agent faster