When an agent gives you a wrong answer, the cause is usually in its files, not in the model. Nate, of the AI Automation channel, runs his business from one big folder of files that Claude reads and writes, and adds new data to it every week. This guide distills the four ways such a knowledge base drifts from reality, and the habits he uses to catch the drift.
01Four ways the files go wrong
Context is whatever material sits in front of the model when it answers. When the answer comes out wrong, one of four things went wrong with that material.
| Failure | What it looks like | The fix |
|---|---|---|
| Poisoning | A false fact sits among the true ones, and the agent relays it as data | Verify: cross-check a live source or a web search, send low-confidence answers to a human |
| Bloat | So much is loaded that the relevant detail drowns | Load less by default; fetch detail on demand (next section) |
| Confusion | A fact is missing or off-topic, so the model fills the gap itself | Close the gap, or have the agent say the data is absent |
| Clash | Two sources disagree, and the agent cannot tell which to trust | Keep one source of truth; date-stamp or retire the old version |
Clash is the sneakiest. In March your policy was always refund; in June it became never refund. Both files still exist, so a refund question gets the March answer, the June answer, or an invention, depending on which file the agent reads first.
Bloat compounds the other three: the more you load, the harder the model works to find what matters. Stop hitting your token limit measures that cost in tokens; this guide measures it in wrong answers.
02Two kinds of context
Everything the agent might need goes in one of two piles.
Expertise context: what every run needs. Who you are, what the business does, the standing policies. It earns a permanent seat in the instructions.
Situational context: what one run needs. Yesterday’s support ticket, one client’s contract. The agent fetches it at the moment the question comes up, uses it, and lets it go.
Nate’s analogy is a principal and a teacher making a seating chart. The principal knows how classrooms should run; the teacher knows which student needs to sit near the board. Rulebook knowledge stays loaded, per-class knowledge gets looked up.
Filing a fact in the wrong pile is how the failures above start. Situational data loaded permanently becomes bloat now and clash later, when the ticket is stale but still in front of the model.
03Turn the root file into a router
CLAUDE.md is the instructions file Claude Code reads at the start of every session, and most
people write it as a system prompt: background, duties, tone.
Nate’s root file is a routing table instead. Two lines of identity, then where things live:
meeting notes here, client work here, API keys here, templates here. Folders deeper in the tree
carry their own CLAUDE.md with project-specific instructions, the behavior-steering kind that
Turn repeated fixes into rules argues for.
The root stays a map, because in a knowledge base of thousands of files, an agent without a map
can spend five minutes failing to find one.
The layout itself matters less than the routing. Nate’s test for whether yours works: pick something you made, open your file explorer, and find it by clicking folders, with no search box. If you can walk the path yourself, an agent with routing rules can too. If you get lost in your own folders, so will it.
04Audit the claims, then approve
A knowledge base makes claims: this index lists the folders, this summary reflects the
meetings, this feed is current. Nate’s habit is a recurring audit, packaged as a skill, that
checks every claim against reality and is forbidden to fix anything. It reads the whole
knowledge base, then writes a report to an audits/ folder and waits for approval.
“Indexes and wikis are claims about what exists and what's current.”
Nate, AI Automation, “5 Hacks to Instantly Level Up Your AI OS” · 10:06The audit runs three checks:
- Routing integrity: does everything the router points to exist, and is anything filed where no route reaches it
- Index truth: do the lists match the disk
- Freshness: when did each data source last update, and is that on schedule or a silent stall
His own run shows what drift looks like. The index claimed 55 folders; the disk had 79. The knowledge was current through June 29, so any July question would get a confident June answer. None of it was visible in daily use, which is the point: drift accumulates silently until an audit dates it. The same rot hits managed memory stores, and Give your agent a memory covers the pruning side.
05Schedule pulls, split what grows
Schedule the pulls. Freshness problems repeat on a calendar, so the fix belongs on a calendar. Nate’s Monday Q&A and Tuesday leadership meeting feed his knowledge base every week, so scheduled runs pull the transcripts in automatically instead of relying on him to remember. Automate a workflow as a Claude routine covers how to set one up in plain language.
Split what grows. Nate kept one master wiki until Claude itself suggested splitting it, because two distinct kinds of data (YouTube transcripts, meeting transcripts) were both growing on their own schedules. Two focused wikis mean the agent searches a fifth of the files, spends fewer tokens, and grabs the wrong document less often.
The rule generalizes: when a distinct pile of knowledge keeps growing, give it its own home and a route in the router. The same instinct, applied for a whole team, is Build a knowledge base your team can ask.
06Backtrack the wrong answer
The last habit runs at the moment of failure: the agent searched five minutes for a file it should have found instantly, or claimed it has no access to data you know it has. The reflex correction (“you do have access, never say that again”) teaches it nothing durable.
Nate instead has it backtrack:
- Retrace: go look at what you did and where you searched.
- Explain: name the wrong turn, the missing route, or the misfiled folder.
- Repair: apply the fix it just proposed, grounded in that evidence.
Each backtrack converts one bad search into one permanent repair, the same loop Turn repeated fixes into rules builds for code.
Further reading
- Nate (AI Automation) · 5 Hacks to Instantly Level Up Your AI OS, the video this guide distills
- Anthropic · Claude Code memory documentation, the CLAUDE.md mechanics the router rides on
- Chroma · Context Rot research, measured evidence that bloat degrades answers even inside large context windows
- ryanportfolio · AI-Firmware template, a ready-made boot layer with a routing kernel and homes for the rules this guide has you write