The Keeper is a librarian for a personal library of Claude Agent Skills. It measures the exact character budget every skill spends in the model’s selection listing, finds which skills silently compete for the same request, and refuses to let a new skill exist without a recorded baseline failure it fixes and a passing before/after comparison proving it helped. It exists because the library’s own eval files turned out to be in a format its test runner couldn’t read, and the live account was already over the undocumented limit that makes a skill quietly stop being found: two defects that had been shipping clean for months because nothing was checking for them.
The problem
A skill library is not self-correcting just because every skill in it passes its own review. Mine had grown to thirteen skills, each one individually reasonable, and I had no way to answer three questions that only get harder to answer as the library grows.
Was any skill earning its place, or just sitting there because
nobody had checked? Every skill shipped with an evals file, and the
actual test runner, claude plugin eval, discovers cases from a
specific folder format. My files were in none of the shapes it
looks for. Twelve skills, sixty-seven written test cases, and the
harness that was supposed to run them found zero.
Was I about to make a skill invisible by adding one more? Claude Code names every enabled skill in the system prompt, and that listing has a hard character ceiling. Past it, a skill isn’t dropped with a warning; it simply stops being named, and the model is told not to reach for anything it wasn’t shown. My library was already at 62.6% of the default limit before I built anything, and I had no instrument that would tell me when the next addition tipped it over.
And when two skills plausibly answered the same request, which one won, and did I even know they were competing? Nothing was measuring it. The scaling research on this is specific: pass-rate loss at library scale runs up to 21%, and up to two-thirds of that loss is exactly this, two skills shadowing each other, not the cost of listing more of them.
Three real risks, each invisible to a validator that checks one skill at a time.
What I built
Not a new way to write a skill. The Keeper is the librarian: a set of deterministic checks that run on the library as a whole, and a discipline that gates every new skill on evidence rather than on how confident I was writing it.
The checks are ordinary Python, on purpose. One measures every description’s character count against the selection ceiling and reports headroom, not just a status. Another runs pairwise similarity across every description in the library and names the shared vocabulary behind a collision, so the fix is obvious rather than a feeling. A library-wide audit rolls both into a single committed health document. Committed on purpose, because a diff between two runs is the fastest way to see a library quietly rotting.
The discipline is three gates, borrowed from the strongest published methodology I could find and made mechanical rather than aspirational:
- Gate A. Say the triage verdict out loud, including “do not build this,” before a file exists. Extending an existing skill is the default outcome, not the fallback.
- Gate B. Run the task with every skill disabled and record what actually goes wrong. If nothing goes wrong, there is no skill here, and that is a successful run.
- Gate C. Run the same task with the skill enabled and compare. At least one check has to pass with it and fail without it. If everything passes both ways, the skill is not earning its place, and the report says so without softening it.
The harder half of the build was the sixty-seven legacy test cases and the harness meant to run them. Another story, another time.
What changed
- The library went from “probably fine” to measured. 71.2% of the selection budget, an exact similarity score on every skill pair, a named list of skills that have never been checked against a baseline: all in one committed file that diffs cleanly, instead of a feeling about whether things still worked.
- A real work order got triaged before it got built. A parallel request proposed nine new skills. Run cold through Gate B, six of the nine reproduced without help (the model already did the thing they would have taught) and were cut before a line of any of them was written. Three survived, narrowed to only what the baseline actually failed at.
- The gate stopped being optional. A CI check now fails the same class of defect the library shipped with for months: a case format the test runner can’t read, a skill that duplicates another’s trigger vocabulary, a budget breach with no headroom left to notice it.
What I’d do differently
I’d sandbox the first baseline run properly the first time. My first attempt at Gate B gave the “no skill installed” control session read access to the very repository whose absence it was supposed to measure, which is exactly the contamination the control is meant to rule out. It found the library instead of failing the task, and the run had to be thrown out and rerun cold, in an empty directory, before the result meant anything.
I’d also assume less about what a compiled tool documents. Two separate defects in this build existed because a written claim about how a piece of tooling behaved went unverified until CI verified it the hard way, in public, on a push.