WikiTrace: Evidence-Linked Memory for Reliable AI Agents

WikiTrace is an evidence-linked memory system for LLM agents. It stores each distilled lesson as an insight connected to a diagnosis of the source trajectory and the raw trajectory itself. At task time, the agent can inspect the evidence when an insight is underspecified, conflicts with another insight, or comes from a failed run. The authors evaluate the approach across eight task suites from five agent benchmarks and report improved average scores and lower memory-induced regression than memory baselines.

External memory is usually introduced as a capability upgrade. An agent solves a task, a memory compiler extracts a reusable rule, and a future agent retrieves that rule to work faster. But the memory can become a source of error.

A lesson may be factually wrong because the source run was misunderstood. It may be overgeneralised from one environment state. It may omit a precondition, a failed intermediate action, or a detail that explains why the original solution worked. Even a correct lesson can be misread or applied to a task that only looks similar.

The paper calls the resulting failure memory-induced degradation: an agent fails a task it would have solved without memory because it followed a faulty or misapplied memory. This is different from ordinary task difficulty. The memory layer itself has made the system worse.

The distinction matters for production systems. If evaluation only measures the success rate with memory enabled, teams can miss tasks where memory actively harms the agent. A reliable evaluation needs both gains and regressions, with a no-memory control.

WikiTrace organises memory into three linked levels.

Insight: a concise, actionable lesson. It records what the agent learned, when the lesson applies, and the identity of the source run.

Compilation: a diagnosis of the source trajectory. It explains why the run succeeded or failed and provides an intermediate layer between a short lesson and a long raw log.

Raw trajectory: the original task, actions, observations, tool calls, and outcome. This is the most detailed evidence layer.

The links create a navigable evidence chain: insight → compilation → raw trajectory. A future agent can start with the compact insight and descend only when more context is needed.

This hierarchical structure is a compromise between two extremes. Keeping only summaries is cheap but can discard crucial evidence. Injecting every raw trace into every prompt is expensive and overwhelms context. WikiTrace keeps the summary as the default interface while preserving evidence for selective inspection.

A practical interpretation of the WikiTrace pipeline has two phases.

Memory construction: collect task trajectories; label each run with its outcome; compile a diagnosis of why it succeeded or failed; distil a concise insight with applicability conditions; and store stable links between the insight, compilation, and raw run.

Memory use: expose the insight set to the next agent; let the agent identify missing details or conflicts; open the compilation and raw trajectory through tools when needed; refine the insight's scope; and act using the corrected understanding.

The repository presents a reproducible workflow organised around building memory, running agents with a selected memory method, and analysing outcomes. This is a research implementation rather than a turnkey managed memory service.

For production, the pipeline should additionally track provenance identifiers, model and prompt versions, environment snapshots, timestamps, tool schemas, data retention policies, and immutable trace checksums.

A practical implementation can use a small relational schema or document store with three linked entities.

Insight record: insight_id, text, applicability, source_run_id, source_outcome, created_at, compiler_version.

Compilation record: compilation_id, source_run_id, diagnosis, success_or_failure_reason, relevant_steps, unresolved_questions.

Trajectory record: run_id, task_id, environment, action_observation_log, tool_calls, final_state, model_version, prompt_version, checksum.

The agent receives a compact insight index and has tools such as open_compilation(insight_id) and open_trajectory(run_id). The tool layer should enforce access controls and return bounded, relevant trace ranges rather than dumping entire histories into the prompt.

This schema is an engineering adaptation of the paper's three-layer concept, not a claim that the authors prescribe these exact field names. In production, store immutable evidence separately from editable summaries so that a revised insight never overwrites the original trace.

The paper compares WikiTrace with a no-memory agent and several memory approaches.

No Memory is the same agent without a learned memory store. It is essential for measuring whether memory helps or harms tasks that the agent could otherwise solve.

AWM (Agent Workflow Memory) distils successful trajectories into reusable procedures.

ExpeL extracts and stores lessons or rules from experience.

ReasoningBank stores strategies from successful and failed experiences and uses task similarity to retrieve relevant lessons.

Insights Only is the most important controlled comparison. It uses the same distilled insights as WikiTrace but removes the evidence links and descent mechanism. This helps isolate the value of access to the compilation and raw trajectory rather than simply comparing different memory content.

The Insights Only baseline makes the paper's core claim more precise: the improvement is not merely that the system has better summaries. It is that the agent can inspect the evidence behind those summaries when it needs to.

Across eight suites, the paper reports the following average scores over three trials:

WikiTrace leads the same-insights-without-evidence baseline by 3.2 points on the reported average. It also exceeds the other memory baselines in this experimental setup.

The average combines different benchmark metrics, so 90.2 should not be interpreted as a universal 90.2% task success rate. It is a cross-suite aggregate reported by the authors. The result is also tied to the specific agent, model, prompts, and task splits in the paper.

The key result is the controlled comparison with Insights Only: preserving the evidence chain adds value beyond storing the distilled lessons alone.

WikiTrace is a promising research prototype, but several limitations matter.

The experiments use a particular Pi terminal-agent and DeepSeek-V4-Flash configuration. Generalisation to other LLMs, coding agents, multi-agent systems, and enterprise workflows remains to be established.

The evaluation covers five benchmarks and eight suites, with three trials. The benchmark average combines different metrics, and some individual results have meaningful variance. Independent reproduction is still needed.

The current design places all insights in the initial agent context. This may become impractical as memory grows. Similarity-based or hierarchical retrieval is an open engineering question.

Evidence links can reveal why a lesson was valid in its source run, but they do not guarantee that the current task satisfies the same conditions. A valid lesson can still be applied to the wrong task if applicability is poorly specified. The authors' discussion recognises the difficulty of writing complete applicability conditions from sparse success/failure feedback.

Finally, raw trajectories may contain sensitive data, secrets, prompt injections, or untrusted tool outputs. Evidence access must be treated as a security-sensitive operation.

WikiTrace makes a strong case for a simple design principle: do not store only what an agent learned; store why it believes the lesson and where the evidence came from.

Its three-layer structure—insight, compilation, raw trajectory—keeps routine memory compact while allowing deeper inspection when a lesson is incomplete, conflicting, or derived from failure. The controlled Insights Only comparison suggests that evidence links add value beyond the same summaries alone.

For production agent builders, the most useful ideas are trace provenance, selective evidence descent, separate measurement of memory gains and regressions, and immutable raw histories. The limitations are equally important: context scaling, evidence retrieval cost, applicability errors, privacy, and the need for independent replication.

The central question for agent memory is shifting from “Can the agent remember?” to “Can the agent verify what it remembers before acting?”

Primary keywords: WikiTrace, evidence-linked memory, LLM agent memory, AI agent memory systems, memory-induced degradation, reliable AI agents.

Technical keywords: trajectory-based memory, memory provenance, episodic memory for LLM agents, experience distillation, source-run retrieval, traceable agent memory, memory verification, agent memory evaluation.

Benchmark keywords: ALFWorld agent memory, ScienceWorld benchmark, PlanCraft benchmark, AppWorld agent, tau3-bench Retail, ReasoningBank, ExpeL, Agent Workflow Memory.

Long-tail queries: how to build reliable memory for AI agents; why LLM agent memory causes failures; how to reduce memory-induced degradation; evidence-based memory for coding agents; how agents verify lessons from past tool calls; persistent memory architecture for autonomous agents.