Agent knowledge base rot: 14 symptoms, checks, and fixes

An agent-maintained knowledge base fails in specific, repeatable ways. This is the reference list: each failure mode below has a symptom you can recognize, a check you can run in minutes, and the smallest fix that works. The modes come from reading the comments on Karpathy's llm-wiki gist, the practitioner reports around it, and maintaining a live agent KB. The story of where each mode showed up in the wild is in the companion post; this page is the one to keep open while you fix yours.

If you'd rather have an agent run all 14 checks for you, there is a read-only audit prompt that returns a scored report card. Get it here.

1. Compounding claims

A wrong entry does not reset like a hallucination. It becomes a prior that every future answer builds on, and agents start citing other agents' unverified entries as evidence.

• Symptom: the agent repeats a claim you never verified. Two pages cite each other and nothing else.
• Check: pick five substantive claims from your most-used pages. Trace each to a raw source or an external reference.
• Fix: agent-written claims land as drafts. A human promotes them. Claims with no source pointer get marked unverified.

2. Duplicate concepts

The same concept lives under two names. Updates land on one twin and the other keeps serving the old version.

• Symptom: search finds two pages that half-answer the question differently.
• Check: scan titles and headings for synonyms of your core concepts.
• Fix: one fact, one home. Keep a glossary of canonical names. Merge twins and leave a pointer at the old location.

3. Silent supersession

New information overwrote the old claim instead of marking it. The contradiction is invisible and history exists only in git, where no agent looks by default.

• Symptom: no page ever disagrees with itself. Decisions look like they were always the current ones.
• Check: find a fact you know changed. Does the page show the old claim as superseded, or did it vanish?
• Fix: supersession callouts: flag the old claim with a dated "superseded" note and keep it visible.

4. External-referent rot

A note describes a system that changed. Lint cannot catch it because the contradiction is not between notes. It is between a note and reality.

• Symptom: a note says the dashboard requires admin access. The permission check stopped requiring it months ago.
• Check: sample five claims about things outside the KB (code, prices, rules, tool behavior). Verify each against the referent.
• Fix: claims about external things carry a pointer to what they describe (file path, URL) plus a date, and the lint pass re-checks the oldest ones.

5. Index drift

The index or map file disagrees with the pages on disk. The agent navigates by the index, so it navigates wrong.

• Symptom: the index lists pages that were renamed or deleted, and misses new ones.
• Check: diff index entries against the file listing.
• Fix: the index updates in the same edit as the page, or is generated, never maintained by memory.

6. Orphan pages

Pages with no inbound links. They exist, cost attention during search, and are never reached by navigation.

• Symptom: pages nobody (human or agent) has opened since creation.
• Check: build the link graph, list zero-inbound pages.
• Fix: link on creation, and let the recurring lint list orphans for merge or deletion.

7. Schema bloat

The instruction file (CLAUDE.md, AGENTS.md) grew past what the agent actually follows. Long instruction files fail wholesale, not gracefully: practitioners report agents ignoring rules entirely past roughly 100 to 300 lines.

• Symptom: the agent breaks rules that are plainly written in the file.
• Check: count lines. Over 150 is a warning. Over 300 is a failure.
• Fix: keep the schema to constraints, not essays. Move detail to linked pages the agent loads on demand.

8. Write-without-read

Everyone is writing, nobody is reading. The KB grows faster than it is used, which is how plausible-but-wrong content accumulates unchallenged.

• Symptom: page count climbs while recent work references the same few pages.
• Check: compare pages created versus pages referenced in the last few weeks of work (git log helps).
• Fix: a promotion gate slows writing. A prune pass in lint removes what nothing referenced.

9. Summaries without backpointers

The KB layer is lossy compression. Summaries drop caveats, dates, exact numbers, and minority views. Once queries hit only the summary, its errors are the knowledge base.

• Symptom: a page states a number or quote with no link to where it came from.
• Check: sample derived pages, look for source pointers.
• Fix: keep an immutable raw layer, and every derived claim links back to its source.

10. One corpus for every task

Relevance is task-relative. A fact that is signal for one task is noise for another, and a single shared corpus feeds every task both.

• Symptom: the agent cites marketing notes while debugging, or project A's conventions leak into project B's answers.
• Check: does one retrieval surface serve unrelated domains?
• Fix: scope by domain or task: separate directories or KBs, routed by what the task needs.

11. No sensitivity split

Secrets, personal data, or NDA material sit in the same corpus as shareable notes, one retrieval away from the wrong context or the wrong model.

• Symptom: client names in the same folder tree as public draft posts.
• Check: grep for the things that must not leak.
• Fix: split by sensitivity. Exclusion rules for cloud agents, local-only zones for what cannot leave the machine.

12. Concurrency clobber

Multiple writers, last write wins. With several agents or several sessions, edits vanish and pages contradict themselves within a day.

• Symptom: conflict markers, duplicated sections, same-day contradictory edits.
• Check: git log for overlapping writes to the same pages.
• Fix: one writer per domain, or explicit turn-taking. Multi-agent setups need typed relations (supersedes, contradicts) instead of plain overwrites.

13. No provenance trail

Nothing records when or why a claim entered. When something smells wrong, there is no way to re-derive it.

• Symptom: a page with no dates, no updated field, no log entry.
• Check: open five pages, look for dates and a log.
• Fix: frontmatter with an updated date, a log file for ingests and decisions, git as the audit trail underneath.

14. Session-memory reliance

Decisions live in chat scrollback instead of files. The next session re-asks, re-derives, and sometimes re-decides differently.

• Symptom: "we decided this last week" is not written anywhere.
• Check: does the schema tell the agent to write decisions to files as they happen?
• Fix: write-as-you-go as a standing rule, and an end-of-session flush for anything that only exists in the conversation.

Running the checks

Checking all 14 by hand takes an afternoon. The audit prompt does it in one pass: paste it into Claude Code, Codex, or Cursor opened in your knowledge base folder, and it returns a report card with a verdict and the smallest fix for everything it flags, read-only. Get the audit prompt. The prevention templates (a hardened CLAUDE.md and a recurring lint checklist) are free on GitHub.