AI Agent Memory: Short-Term vs Long-Term, and How to Actually Implement It (2026)

What is AI agent memory, and why isn't a context window enough?

An AI agent runs a loop — perceive, plan, act, observe — and every one of those steps needs to know what happened before. The context window is where that information lives during a single call, but it is bounded, and it is not memory in any durable sense: close the session, or push enough new turns through it, and anything not actively re-sent is gone. That's the problem agent memory solves — giving an agent a way to keep and recall information that outlives any single context window.

The useful split, used across the tools in this guide, is short-term memory (how you manage what's inside the current context window) versus long-term memory (what's persisted outside it and pulled back in only when relevant). A big context window makes the short-term side more forgiving, but it doesn't replace the long-term side — an agent with a 1M-token window still has no way to recall a fact from a session that ended yesterday unless something wrote that fact down somewhere it can search later.

Short-term memory: managing the context window

Short-term memory is about keeping the current conversation inside the model's context limit without losing what matters. Three patterns do most of the work:

All three are still working inside the context window — they decide what's in it, not where information goes once it falls out. That's where long-term memory comes in.

Long-term memory: vector retrieval, knowledge graphs, and hybrids

Long-term memory persists facts outside the context window and retrieves only what's relevant to the current turn, the same retrieve-then-inject shape as RAG. Two storage approaches dominate:

The common thread across all of them: write happens continuously as the agent runs (not as a one-time ingest like a RAG corpus), and read happens selectively — only the facts relevant to the current turn get pulled back into the context window.

Mem0 vs Letta vs Zep vs LangMem: how they compare

These four show up constantly in agent-memory discussions, and they take genuinely different bets on who decides what gets remembered.

Mem0 is a library/server you call around an existing LLM app — install it, call add() after a turn and search() before the next response, and an extraction pipeline decides what's worth keeping (Mem0 GitHub README). Letta, the productized successor to the MemGPT research project, instead gives the agent itself the tools to edit its own memory blocks and decide what stays in context vs. gets paged to an archival store (Letta GitHub repository). Zep's differentiator is time: its open-source Graphiti engine builds a bi-temporal knowledge graph so facts that change don't just get overwritten — the history is queryable (Graphiti documentation). LangMem is the lightest-weight of the four if you're already in the LangChain/LangGraph ecosystem: it's a set of memory primitives and tools (create_manage_memory_tool, create_search_memory_tool) wired into LangGraph's existing long-term memory store rather than a separate service (LangMem GitHub repository; LangChain docs — long-term memory).

How to implement a basic memory layer

You don't need any of the tools above to get the core pattern working. A minimal long-term memory layer is three functions: extract facts from a turn, store them with embeddings, and retrieve the relevant ones before the next response. Here it is with Chroma, the same way our vector database for RAG guide builds the retrieval side:

# A minimal long-term memory layer: embed, store, retrieve.
# Swap chromadb for Mem0/Letta/Zep's SDK and the shape is the same.
import chromadb
from openai import OpenAI

client = chromadb.PersistentClient(path="./agent_memory")
memory = client.get_or_create_collection(name="user_facts")
llm = OpenAI()

def extract_facts(conversation_turn: str) -> list[str]:
    # One LLM call: pull out durable facts worth remembering.
    # ("User prefers concise answers", "User's timezone is IST", ...)
    resp = llm.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{
            "role": "user",
            "content": f"Extract durable facts worth remembering, one per line:\n{conversation_turn}",
        }],
    )
    return [f.strip("- ") for f in resp.choices[0].message.content.splitlines() if f.strip()]

def remember(conversation_turn: str, user_id: str):
    for fact in extract_facts(conversation_turn):
        memory.add(
            ids=[f"{user_id}-{hash(fact)}"],
            documents=[fact],
            metadatas=[{"user_id": user_id}],
        )

def recall(query: str, user_id: str, k: int = 3) -> list[str]:
    results = memory.query(query_texts=[query], n_results=k, where={"user_id": user_id})
    return results["documents"][0]

# At response time: pull relevant facts into the system prompt before calling the LLM.
relevant = recall("What does this user prefer?", user_id="u1")
system_prompt = "Known facts about the user:\n" + "\n".join(relevant)

Three things worth noting. Extraction is its own LLM call — you're not embedding the raw conversation, you're asking a model to decide what's durable first, which is exactly what Mem0's pipeline does under the hood. Writes are scoped — every fact carries a user_id in its metadata, the same pattern pgvector or any filtered vector search uses, so one user's facts never leak into another's retrieval. Retrieval happens before generation, not during it — the facts get folded into the system prompt, not fetched by the model mid-answer, which keeps the pattern simple to reason about and test.

Moving from this to Mem0, Letta, or Zep mainly swaps who does the extraction and what the store looks like underneath — a graph instead of flat vectors for Zep, agent-driven tool calls instead of a background pipeline for Letta — but the write → embed/structure → retrieve → inject loop stays the same.

When do you actually need long-term memory vs RAG/context alone?

See our RAG vs fine-tuning guide for the adjacent decision of whether to retrieve knowledge at all versus bake it into model weights — memory is a third axis on top of that, about what persists from the agent's own interactions rather than from a static corpus.

Frequently Asked Questions

References

• • Mem0 — GitHub repository and README
• • Letta (formerly MemGPT) — GitHub repository
• • Rasmussen et al. — "Zep: A Temporal Knowledge Graph Architecture for Agent Memory" (arXiv, 2025)
• • Graphiti — official documentation
• • LangMem — GitHub repository
• • LangChain docs — long-term memory
• • LangChain — LangMem SDK launch post (memory types)
• • Chroma — documentation