Giving an Agent Memory Without Giving It Amnesia

What agent memory actually is

Teams usually arrive at memory after a demo where the agent forgot something obvious, and the instinct is to add a vector store and write everything into it. That produces a system which remembers a great deal and can be trusted about none of it. The useful question is not how to remember more. It is which store each fact belongs in, how long it stays true, and what happens when it is wrong.

Four stores, four lifetimes

Write the four stores down before you choose a database. The technology argument is downstream of this table, and most disagreements about vector stores dissolve once people notice they are describing different rows.

The Memory Write Test

Four questions, asked before any write. They take a minute each in design and they remove the class of bug where a guess quietly becomes a permanent fact about a customer.

The fourth question is the one that turns into an incident report if it is skipped. Two accounts sharing a memory index with no subject key on the record will eventually surface one customer's constraint inside another customer's conversation, and the retrieval log will make it look like the model invented it.

What summarisation quietly destroys

Compaction is lossy compression, and the loss is not random. It systematically removes the things that made a case difficult and keeps the things that were repeated often, which is close to the opposite of what you want.

• Negations invert or vanish. A customer who does not want a replacement becomes a customer who discussed a replacement. This is the single most damaging loss, because the summary now reads as mild permission.
• Numbers and identifiers blur. Exact amounts become roughly, and reference numbers get dropped as noise because they carry no semantic weight.
• Constraints soften into preferences. A hard requirement written by a human becomes something the customer would like, and the agent treats it as negotiable.
• Provenance disappears entirely. After one pass you cannot tell what the person said from what the model concluded, and after two passes nobody will ever be able to.
• Rare detail is discarded first. The unusual fact is exactly what made this case need a human, and it is the most likely thing to be compressed away.

Retrieval, or memory you cannot find

A fact you cannot retrieve at the right moment is not memory, it is storage. Most memory systems that disappoint in production have a retrieval problem rather than a writing problem, and the cause is usually reaching for semantic search where a key lookup was correct.

1. Key first, search secondThe ordering that fixes most systems

   If you know the account, fetch its records by key. Embedding search over short factual statements is weak, because two sentences about different customers with the same structure sit close together in vector space.

2. Inject records, not proseStructure survives attention better

   Give the model a small list of typed records with statement, source, written date and confidence, rather than a paragraph of remembered narrative. It can then weigh them, and you can log exactly what was injected.

3. Cap the number injectedMore is worse

   A handful of relevant records outperforms thirty, because every extra record competes for attention with the instruction that mattered. Set a hard cap and let relevance ranking fight for the slots.

4. Always show age and sourceOne line per record

   A model told a preference was recorded fourteen months ago by an inference treats it differently from one stated last week by the customer. Without those fields, everything reads as equally true.

5. Log the retrieved set with the runFor the question you will be asked

   Why did it say that is answered by what was retrieved, not by the prompt. If the retrieved set is not in the trace, the behaviour is unexplainable. The same reasoning applies to grounded answers in RAG explained for builders.

One more rule with a cost attached: retrieved memory is charged on every turn it stays in context. Injecting fifteen hundred tokens of profile on a twelve turn run is eighteen thousand billed tokens, most of them irrelevant after turn two. Inject what the current step needs, and refresh rather than accumulate.

Memory poisoning and the reinforcement loop

A wrong fact written once does not stay one wrong fact. It gets retrieved, repeated, folded into a summary, and eventually read by a human who assumes a person verified it. The path is predictable enough to draw, which is what makes it preventable.

The context budget arithmetic

Memory competes for the same space as tool schemas, instructions and history, and the competition is arithmetic rather than opinion. Work out the budget before deciding how much profile to inject, because the answer is usually smaller than the design assumed.

Two warnings about reading those outputs. Filling the window is not the goal and behaviour degrades well before it, so treat anything above roughly half as a design smell rather than headroom. And the number that matters is not the total, it is how much of the context is still about the current task by the final turn. That ratio, not the window size, is what people are describing when they say a long run went strange. More on the limits in context windows explained.

A record schema and a write policy

Memory becomes manageable the moment it stops being free text and starts being rows with required fields. The schema below carries the four answers from the write test, and the policy underneath it is the part that goes into the agent's own prompt.

// memory_record.v1  -  one row per remembered fact
{
  "$id": "memory_record.v1",
  "type": "object",
  "additionalProperties": false,
  "required": ["id","scope","subject_key","kind","statement",
               "source","confidence","written_at","valid_until"],
  "properties": {
    "id":          { "type": "string" },
    "scope":       { "enum": ["run","thread","account","user"] },
    "subject_key": { "type": "string",
                     "description": "tenant, account or user key. Every query filters on this. Never optional." },
    "kind":        { "enum": ["stated_fact","tool_fact","model_inference","constraint","preference"] },
    "statement":   { "type": "string", "maxLength": 240 },
    "negation":    { "type": "boolean",
                     "description": "true when the statement records something that must NOT happen. Records with negation true are never compacted." },
    "source": {
      "type": "object",
      "required": ["type","ref"],
      "properties": {
        "type": { "enum": ["user_message","tool_result","human_edit","model_inference"] },
        "ref":  { "type": "string", "description": "message id, trace id or run id. Must resolve in six months." }
      }
    },
    "confidence":     { "type": "number", "minimum": 0, "maximum": 1 },
    "written_at":     { "type": "string", "format": "date-time" },
    "valid_until":    { "type": "string", "format": "date-time",
                        "description": "required. A fact with no expiry becomes folklore." },
    "superseded_by":  { "type": ["string","null"] }
  }
}

/* WRITE POLICY  -  paste into the agent's system prompt
 - Write a memory only if the fact will still matter in a later run.
 - Never write kind model_inference with confidence above 0.6.
 - Anything the system of record already holds is fetched, never remembered.
 - A statement containing a negation is written with negation true.
 - Every write names a valid_until. If you cannot choose one, use 90 days.
 - Before writing, search for an existing record on the same subject and
   supersede it rather than adding a second version.
*/

Terms used precisely here

Working context

Everything present in the message list for the current run: the system prompt, tool schemas, the history so far, and anything retrieved. It is the only thing the model can see, and it is discarded when the run ends.

Episodic memory

A record of what happened in previous runs, stored as events or transcripts and retrieved by identifier or search. It answers what happened before, not what is currently true.

Semantic memory

Durable statements about an account or person that outlive any single run, such as constraints and preferences. Each one needs a source, a subject key and an expiry, or it decays into an unattributable claim.

Context compaction

Replacing part of a run's history with a shorter summary to free space. It reliably preserves what was repeated and reliably loses negations, identifiers and provenance, so anything load-bearing must be excluded from it.

Memory poisoning

The process by which one incorrect written fact is retrieved, restated and folded into summaries until it is treated as verified, with the original error no longer traceable.

Do this in the order above and memory stays a small, boring subsystem with rows you can read. Do it in the usual order, which is to add a vector store and see what happens, and it becomes the component nobody can reason about six months in. The loop it plugs into is described in what an AI agent actually is, and building both properly is AI agent development.