TL;DR: A language model remembers nothing between API calls - its only "memory" is whatever text is in the current context window. Agent memory is the engineering around that limit: scratchpads for the current task, summaries that survive a session, and long-term stores the agent can write to and search later.
How it works
Start from the constraint: every model call is stateless. The illusion of a continuing conversation is produced by resending the transcript with each request, and the transcript can only grow until it hits the context window limit. For a long-running agent - hundreds of tool calls, sessions spanning days - "just resend everything" fails twice: the transcript will not fit, and even before it overflows, a context stuffed with stale detail degrades the model's attention on the current step.
Memory systems answer with layers, loosely mirroring working versus long-term memory. Short-term memory is the context itself, actively managed: when the transcript nears the limit, the harness compacts it - older turns are summarized into a paragraph, tool outputs are truncated, and the compressed history replaces the original. The agent keeps the gist and loses the bulk. Many agents also keep a scratchpad - a notes file or plan document they update as they work, re-reading it to reorient after compaction.
Long-term memory lives outside the model entirely, in ordinary storage. Persistent instruction files carry durable preferences ("this user deploys with script X, never Y") that load at session start. Episodic stores hold records of past sessions - what was tried, what worked - that the agent can consult when a similar task recurs. At scale these stores get a search layer: memories are embedded and retrieved by similarity, which is RAG pointed at the agent's own history instead of your documents, often backed by the same vector database machinery.
The interesting design decisions are writes, not reads. What deserves remembering? Verbatim transcripts are cheap to keep and mostly noise; distilled facts are useful but require the agent (or a background process) to decide what to extract. Memories go stale - the user changes deployment scripts, and last month's "always use X" becomes actively harmful - so good systems date, update, and expire entries rather than only appending. The failure modes are memorable for the wrong reason: an agent that confidently applies an obsolete memory is worse than one that asks again.
Where it sits in the AI stack
Memory is the feedback path in the agent stack - past work flows out to storage and back into future context:
Memory quality compounds: an agent that records what it learned makes the next session cheaper and sharper, while one that records noise pays context-window rent on it forever.
Key tools and implementations
-
Instruction files
Persistent markdown files of preferences and project facts loaded at the start of every session.
-
Context compaction
Harness features that summarize older turns in place when the transcript nears the window limit.
-
Memory services
Dedicated stores that extract, embed, and retrieve facts from conversations across sessions.
-
Knowledge-graph memory
Entity-and-relation stores that keep facts structured and updatable rather than as loose text.
Related entries
- Context window The maximum amount of text, measured in tokens, that a model can consider in a single request.
- AI agent A system where a language model plans, calls tools, and loops on results to finish a task with minimal supervision.
- RAG (retrieval augmented generation) Fetching relevant documents at query time so a language model can answer from your data instead of memory alone.
- Vector database A database that stores embeddings and finds nearest neighbors fast, powering semantic search and RAG.