---
title: "Your agent's knowledge base is lying to you. Run these 14 checks."
description: "The 14 ways an agent-maintained knowledge base rots: each failure mode with a symptom to recognize, a check to run in minutes, and the smallest fix that works."
canonical: https://omiid.me/notebook/67/your-agent-s-knowledge-base-is-lying-to-you-run-these-14-checks
author: Omid Sayfun
published: 2026-08-18
updated: 2026-08-18
---

# Your agent's knowledge base is lying to you. Run these 14 checks.

An agent-maintained knowledge base fails in specific, repeatable ways.
Each failure mode below has a symptom you can
recognize, a check you can run in minutes, and the smallest fix that
works. The modes come from reading the comments on
[Karpathy's llm-wiki gist](https://gist.github.com/karpathy/442a6bf555914893e9891c11519de94f),
the practitioner reports around it, and maintaining a live agent KB. The
story of where each mode showed up in the wild is in
[the companion post](https://omiid.me/notebook/66/i-read-all-1-061-comments-on-karpathy-s-llm-wiki-gist);
this page is the one to keep open while you fix yours.

If you'd rather have an agent run all 14 checks for you, there is a
read-only audit prompt that returns a scored report card.
[Get it here](https://omiid.me/kb-audit?utm_source=blog&utm_content=post-67).

## 1. Compounding claims

A wrong entry does not reset like a hallucination. It becomes a prior
that every future answer builds on, and agents start citing other
agents' unverified entries as evidence.

- **Symptom:** the agent repeats a claim you never verified. Two pages
  cite each other and nothing else.
- **Check:** pick five substantive claims from your most-used pages.
  Trace each to a raw source or an external reference.
- **Fix:** agent-written claims land as drafts. A human promotes them.
  Claims with no source pointer get marked unverified.

## 2. Duplicate concepts

The same concept lives under two names. Updates land on one twin and
the other keeps serving the old version.

- **Symptom:** search finds two pages that half-answer the question
  differently.
- **Check:** scan titles and headings for synonyms of your core
  concepts.
- **Fix:** one fact, one home. Keep a glossary of canonical names.
  Merge twins and leave a pointer at the old location.

## 3. Silent supersession

New information overwrote the old claim instead of marking it. The
contradiction is invisible and history exists only in git, where no
agent looks by default.

- **Symptom:** no page ever disagrees with itself. Decisions look like
  they were always the current ones.
- **Check:** find a fact you know changed. Does the page show the old
  claim as superseded, or did it vanish?
- **Fix:** supersession callouts: flag the old claim with a dated
  "superseded" note and keep it visible.

## 4. External-referent rot

A note describes a system that changed. Lint cannot catch it because
the contradiction is not between notes. It is between a note and
reality.

- **Symptom:** a note says the dashboard requires admin access. The
  permission check stopped requiring it months ago.
- **Check:** sample five claims about things outside the KB (code,
  prices, rules, tool behavior). Verify each against the referent.
- **Fix:** claims about external things carry a pointer to what they
  describe (file path, URL) plus a date, and the lint pass re-checks
  the oldest ones.

## 5. Index drift

The index or map file disagrees with the pages on disk. The agent
navigates by the index, so it navigates wrong.

- **Symptom:** the index lists pages that were renamed or deleted, and
  misses new ones.
- **Check:** diff index entries against the file listing.
- **Fix:** the index updates in the same edit as the page, or is
  generated, never maintained by memory.

## 6. Orphan pages

Pages with no inbound links. They exist, cost attention during
search, and are never reached by navigation.

- **Symptom:** pages nobody (human or agent) has opened since
  creation.
- **Check:** build the link graph, list zero-inbound pages.
- **Fix:** link on creation, and let the recurring lint list orphans
  for merge or deletion.

## 7. Schema bloat

The instruction file (CLAUDE.md, AGENTS.md) grew past what the agent
follows. Long instruction files fail wholesale, not
gracefully: practitioners report agents ignoring rules entirely past
roughly 100 to 300 lines.

- **Symptom:** the agent breaks rules that are plainly written in the
  file.
- **Check:** count lines. Over 150 is a warning. Over 300 is a
  failure.
- **Fix:** keep the schema to constraints, not essays. Move detail to
  linked pages the agent loads on demand.

## 8. Write-without-read

Everyone is writing, nobody is reading. The KB grows faster than it
is used, which is how plausible-but-wrong content accumulates
unchallenged.

- **Symptom:** page count climbs while recent work references the same
  few pages.
- **Check:** compare pages created versus pages referenced in the last
  few weeks of work (git log helps).
- **Fix:** a promotion gate slows writing. A prune pass in lint
  removes what nothing referenced.

## 9. Summaries without backpointers

The KB layer is lossy compression. Summaries drop caveats, dates,
exact numbers, and minority views. Once queries hit only the summary,
its errors are the knowledge base.

- **Symptom:** a page states a number or quote with no link to where
  it came from.
- **Check:** sample derived pages, look for source pointers.
- **Fix:** keep an immutable raw layer, and every derived claim links
  back to its source.

## 10. One corpus for every task

Relevance is task-relative. A fact that is signal for one task is
noise for another, and a single shared corpus feeds every task both.

- **Symptom:** the agent cites marketing notes while debugging, or
  project A's conventions leak into project B's answers.
- **Check:** does one retrieval surface serve unrelated domains?
- **Fix:** scope by domain or task: separate directories or KBs,
  routed by what the task needs.

## 11. No sensitivity split

Secrets, personal data, or NDA material sit in the same corpus as
shareable notes, one retrieval away from the wrong context or the
wrong model.

- **Symptom:** client names in the same folder tree as public draft
  posts.
- **Check:** grep for the things that must not leak.
- **Fix:** split by sensitivity. Exclusion rules for cloud agents,
  local-only zones for what cannot leave the machine.

## 12. Concurrency clobber

Multiple writers, last write wins. With several agents or several
sessions, edits vanish and pages contradict themselves within a day.

- **Symptom:** conflict markers, duplicated sections, same-day
  contradictory edits.
- **Check:** git log for overlapping writes to the same pages.
- **Fix:** one writer per domain, or explicit turn-taking. Multi-agent
  setups need typed relations (supersedes, contradicts) instead of
  plain overwrites.

## 13. No provenance trail

Nothing records when or why a claim entered. When something smells
wrong, there is no way to re-derive it.

- **Symptom:** a page with no dates, no updated field, no log entry.
- **Check:** open five pages, look for dates and a log.
- **Fix:** frontmatter with an updated date, a log file for ingests
  and decisions, git as the audit trail underneath.

## 14. Session-memory reliance

Decisions live in chat scrollback instead of files. The next session
re-asks, re-derives, and sometimes re-decides differently.

- **Symptom:** "we decided this last week" is not written anywhere.
- **Check:** does the schema tell the agent to write decisions to
  files as they happen?
- **Fix:** write-as-you-go as a standing rule, and an end-of-session
  flush for anything that only exists in the conversation.

## Running the checks

Checking all 14 by hand takes an afternoon. The audit prompt does it in
one pass: paste it into Claude Code, Codex, or Cursor opened in your
knowledge base folder, and it returns a report card with a verdict and
the smallest fix for everything it flags, read-only.
[Get the audit prompt](https://omiid.me/kb-audit?utm_source=blog&utm_content=post-67).
The prevention templates (a hardened CLAUDE.md and a recurring lint
checklist) are free on
[GitHub](https://github.com/iamomiid/agent-kb-audit).
