Long-Term Memory with Vector Stores

Duration: 15 min

Long-Term Memory with Vector Stores

Duration: 15 min

Overview

Long-term memory gives agents the ability to remember information across sessions - user preferences, learned facts, past decisions, and accumulated knowledge. Vector stores are the dominant technology for implementing this: they convert text into numerical embeddings and enable fast semantic search to retrieve relevant memories.

By the end of this module, you will understand how embeddings work for memory, how to choose a vector store, and how to implement retrieval-augmented memory for your agents.

Key Concepts & Foundations

  • What: Long-term memory persists information indefinitely using vector embeddings for semantic retrieval
  • Why: Agents need to recall relevant past interactions without loading entire history into context
  • How: Embed memories as vectors, store in a vector database, retrieve by semantic similarity

How Vector Memory Works

1. Encode - Convert text into a dense vector using an embedding model 2. Store - Save the vector + metadata in a vector database 3. Query - When the agent needs memory, embed the query and find nearest vectors 4. Retrieve - Return the original text associated with the closest matches

Vector Store Options

| Store | Type | Best For | |-------|------|----------| | ChromaDB | Embedded | Prototyping, single-machine agents | | Pinecone | Cloud | Production, managed infrastructure | | Weaviate | Self-hosted/Cloud | Complex schemas, hybrid search | | pgvector | PostgreSQL extension | Teams already using PostgreSQL | | FAISS | In-memory library | High-performance, read-heavy workloads |

Hands-On Implementation

import hashlib
import math
from dataclasses import dataclass, field
from datetime import datetime

@dataclass
class MemoryRecord:
    """A single long-term memory entry."""
    id: str
    content: str
    embedding: list[float]
    metadata: dict = field(default_factory=dict)
    created_at: str = field(default_factory=lambda: datetime.now().isoformat())

def simple_embedding(text: str, dimensions: int = 64) -> list[float]:
    """Deterministic pseudo-embedding for demo (replace with real model)."""
    words = text.lower().split()
    vector = [0.0] * dimensions
    for word in words:
        h = hashlib.md5(word.encode()).hexdigest()
        for i in range(dimensions):
            vector[i] += int(h[i % len(h)], 16) / 16.0
    magnitude = math.sqrt(sum(x * x for x in vector))
    if magnitude > 0:
        vector = [x / magnitude for x in vector]
    return vector

def cosine_similarity(a: list[float], b: list[float]) -> float:
    dot = sum(x * y for x, y in zip(a, b))
    mag_a = math.sqrt(sum(x * x for x in a))
    mag_b = math.sqrt(sum(x * x for x in b))
    if mag_a == 0 or mag_b == 0:
        return 0.0
    return dot / (mag_a * mag_b)

class VectorMemoryStore:
    """Simple vector store for agent long-term memory."""
    def __init__(self, embedding_fn=simple_embedding):
        self.memories: list[MemoryRecord] = []
        self.embed = embedding_fn
    def store(self, content: str, metadata: dict = None) -> str:
        memory_id = hashlib.sha256(
            f"{content}{datetime.now().isoformat()}".encode()
        ).hexdigest()[:12]
        record = MemoryRecord(
            id=memory_id, content=content,
            embedding=self.embed(content), metadata=metadata or {}
        )
        self.memories.append(record)
        return memory_id
    def search(self, query: str, top_k: int = 3) -> list[tuple[MemoryRecord, float]]:
        query_embedding = self.embed(query)
        scored = []
        for memory in self.memories:
            score = cosine_similarity(query_embedding, memory.embedding)
            scored.append((memory, score))
        scored.sort(key=lambda x: x[1], reverse=True)
        return scored[:top_k]


Demo: Agent that remembers user facts across sessions

store = VectorMemoryStore()
store.store("User prefers Python over JavaScript", {"type": "preference"})
store.store("User works at a startup building ML pipelines", {"type": "fact"})
store.store("User completed the RAG systems course", {"type": "progress"})
store.store("User had trouble with async/await in Python", {"type": "struggle"})
store.store("User is interested in deploying models to AWS", {"type": "interest"})
query = "Help me deploy a machine learning model"
results = store.search(query, top_k=3)
print(f"Query: '{query}'")
print(f"Retrieved {len(results)} relevant memories:")
for record, score in results:
    print(f"  [{score:.3f}] {record.content}")

Advanced Techniques

Building production-grade vector memory systems requires attention to:

1. Metadata Filtering - Combine vector search with metadata filters (user, time range, type) 2. Hybrid Search - Combine semantic similarity with keyword matching (BM25 + vector) 3. Memory Deduplication - Detect and merge similar memories to prevent redundancy 4. TTL and Expiration - Set time-to-live on memories that become stale 5. Embedding Updates - Re-embed memories when switching to a better model

Quiz

Q1: Why are vector embeddings used for long-term memory instead of keyword search?

  • A) They are cheaper to compute
  • B) They capture semantic meaning, finding relevant memories even without exact word matches ✓
  • C) They use less storage space
  • D) They are faster than any other approach

Q2: What is the role of cosine similarity in vector memory retrieval?

  • A) It compresses the vectors
  • B) It measures how semantically similar two pieces of text are ✓
  • C) It encrypts the memories
  • D) It determines the storage location

Q3: When should you use metadata filtering with vector search?

  • A) Never - vector search alone is always sufficient
  • B) When you need to scope results by user, time, type, or other structured attributes ✓
  • C) Only when the vector store is empty
  • D) Only for real-time applications

← Back ▶ Run in Colab