Vol 2, Issue 7 | Feb 18 2026
Your AI Setup Is Quietly Degrading. Here's How to Check.
Dear Reader,
I renamed a skill in my AI setup last week with no issues — no error, no warning, everything looked fine. Go me 👊🏻
Then about three days later, I discovered that separate components of my workflow were silently failing, still referencing the old skill name, quietly returning nothing and skipping tasks without complaint. Silently drifting away from each other.
How AI Setups Can Fail
AI tools work from the context you gave them — not from what's currently true, but from what was true when you set things up. That context doesn't update automatically. The AI doesn't know it's stale.
So when something shifts — a file moves, a tool gets replaced, a workflow changes — the AI keeps working from what it has. It doesn't error. It doesn't warn you. It quietly uses the outdated reference, or skips the step it can't complete, or generates output from a context that no longer reflects reality.
The output still looks plausible. That's the problem.
Silent drift is what I call it when your AI setup is quietly pointing to things that have moved, changed, or disappeared — and nothing tells you. The AI works with what it has. You assume it's working as intended.
It isn't.
This Isn’t Just a Power User Problem
My setup might be unusual — 200+ custom components, all referencing each other by name and file path. But the underlying pattern shows up in much simpler configurations.
Have you ever set up a custom GPT with a system prompt that references “our current pricing” or “our product lineup” — and then never updated it when those changed? Configured an AI assistant to your role, your team, your workflow — and then changed any of those things? Used a saved workflow that was calibrated months ago to conditions that have since shifted?
The AI doesn’t know things changed — it can’t know. It works with the context you gave it when you configured it. If that context has drifted from reality, the output has too. Quietly, without error.
What I Found in My Own Setup
When I actually audited my configuration, the first run found 17 critical issues. Not “could be improved” — critical.
A voice profile for my newsletter agent was pointing to a file I’d moved months earlier. The agent was still running and generating content, just without any of the voice guidance it was supposed to follow. The output looked like mine. It wasn’t quite.
Seven components were referencing tools from a server I’d replaced with something better. They weren’t failing with an error — they were just quietly not invoking the tools correctly.
Three skill references had one-character typos — close enough that I’d never spotted them in the file names, different enough that nothing could find them. Complete failure to invoke. Completely silent.
Plus 55 configuration drift warnings where components were still using patterns from three months ago.
All of it silent. Zero error messages.
Auditing Your Own Setup
The diagnostic logic that uncovered this follows a simple principle: map what you have against what you’re referencing, and find the gaps.
For most AI users, that’s less technical than it sounds. A few questions:
When did you last review your context? Any AI tool you’ve configured with persistent context — custom GPTs, saved instructions, Claude Projects, workflow automations — carries a snapshot of your situation from whenever you set it up. How old is that snapshot?
What’s changed since you configured it? Your role, your team, your products or pricing, the tools you use, the workflows you follow. Any of these can create drift between what the AI was calibrated to and what’s actually true now.
If something was quietly failing, would you notice? The output might still look plausible. The task might still seem to complete. Drift reveals itself gradually — in output that’s a bit off, in steps that get skipped, in advice that doesn’t quite fit anymore.
What are you trusting the AI to remember that it might have gotten wrong? Memory, context documents, saved instructions — these are snapshots. They don’t update automatically.
Three Things To Try This Week
- Audit your most-used AI setup. Pick the workflow you use most — custom GPT, Claude Project, saved prompt, automation — and ask: when did I last review the context I gave it? What’s changed since then?
- Do a “what’s still true” check. Go through the instructions or context you gave your AI tool and mark anything that might no longer be accurate. Outdated role descriptions, old team structures, pricing or products that changed — these are the silent drift candidates.
- Ask the AI to flag its own assumptions. “What are you assuming about me or my situation that might no longer be accurate?” It won’t catch everything, but the answers are often more useful than you’d expect.
For anyone running a custom Claude Code setup with a lot of components: I built a diagnostic tool that automates this. It runs 7 checks — vault path validation, skill cross-references, MCP server health, CLI availability, configuration drift, staleness detection, orphan detection. Open source, MIT licensed. Full breakdown on the blog and on GitHub.
Signal Over Noise Is Going Paid
Quick reminder: Signal Over Noise moves to a paid newsletter on March 4th.
If you’re reading this, you’re already here — and the Founding Member rate of €49/year is still available. That’s not “first year” pricing. It’s your rate, permanently, for being early. Monthly pricing gets announced next week.
The short version of why: I’d rather be accountable to readers than to advertisers. The full explanation is in the announcement post.
Elsewhere
Claude Code Ecosystem Healthcheck
Open sourced this week - Detect drift between interconnected Claude Code components: skills, agents, MCP servers, vault paths, CLI tools, and configuration policies. GitHub.
Claude Code Ecosystem Health Check
The technical deep-dive behind today’s topic — what each of the 7 diagnostic checks actually does, and why grepping for broken references misses what structured scanning catches. For Claude Code users with complex setups. On the blog.
Second Brain Chronicles
New newsletter launched last week - Second Brain Chronicles, where I discuss my Claude Code + Obsidian system more in depth for those who want more of a technical focus. Free each week on Fridays.
Until next week,
Jim
