Agent skills that run a nonprofit, with the grading measured

The problem

A small nonprofit has the same recurring work as a larger company: branded media for every post, demos for every project, résumé reviews for every graduate. Vets Who Code does not have the staff for that. Packaging each job as an agent skill lets one engineer do it from a terminal, the same way every time.

The résumé scorer raised a harder problem. Its scripts are deterministic, and the step where a model reads a résumé and grades each skill is not. If two sessions grade the same résumé differently, the advice built on that score is noise.

Success criteria

The target was written down before measuring: scores from independent grading sessions on the same résumé and the same market data must fall within ±2 points of each other. The lift a single skill adds to a score is 1 to 6 points, so a wider spread would make the ranking of recommendations meaningless.

Approach and architecture

Each skill is a Markdown instruction file with scripts behind it. A shared TypeScript library validates brand packs and checks fonts. The résumé scorer fetches live postings from five public job board APIs plus O*NET and BLS data, a model grades the résumé against that market, and a script computes the score.

Data model

• postings.json: fetched postings with a fetch date. Refused if older than 14 days.
• resume.json: one grade per skill (proven, applied, listed, stale or none), each with the quoted line it rests on.
• report.json: the composite score, weighted 50% postings, 35% O*NET technology skills, 15% O*NET core skills.

The eval

Three résumés in three lanes, each graded by three fresh Claude Code sessions with no shared context. Each session received only the skill file, the checklist, the résumé and the fetched market data. Two résumés are fictional fixtures that plant the hard cases on purpose: a role that ended just over five years ago, a tool the person is only “learning”, bullets with a count and no outcome.

Spread is the highest score minus the lowest across the three sessions, in points out of 100. By round 3 every lane was inside the target.

Where AI got it wrong

Round 1 showed the model being generous in ways that would have misled a job seeker:

• It inferred skills the résumé never named. “Fixed N+1 queries” became SQL. “On ECS” became AWS. An LLM app became AI agents.
• It could not agree on what counts as a metric. Is “40 dashboards” a result or only a count?
• It disagreed about which sections count as experience. A meetup demo under Projects was experience in one session and not in another.

Each disagreement became a written rule. The quote must name the skill. Any number in the best named bullet makes it proven. Experience means a bullet under a dated role or a project. The software lane then got worse in round 2 before it got better, which is why there is a round 3.

The eval also caught a plain bug: a data file from before a vocabulary rename crashed the scorer. The script now refuses that file and asks for a re-fetch.

Key tradeoffs

Strict naming over generous inference

Requiring the quote to name the skill means a person who did the work and described it loosely scores lower. That is the right failure. The score tells them to name the skill, and it never credits them with one they would have to defend. Recorded as DR-009.

Refuse stale data

Postings older than 14 days and O*NET data older than 30 are refused. The user re-fetches more often and never gets a score against last quarter’s market. Recorded as DR-008.

Engineering signals

• CI runs the Node tests, two Python test suites, brand-pack validation, a font check and claude plugin validate --strict.
• Dependencies are pinned to exact versions.
• Score bands are percentiles of a reference pool, and no band is shown with fewer than 20 résumés in it.
• The scorer refuses to read a résumé stored inside a git repository.

AI collaboration note

The skills are instructions for Claude Code, and Claude Code sessions were the graders in the eval. I wrote the rules, designed the fixtures to break them, read every split, and decided which rule each one became.

Known limits

• “Different graders” means different sessions of the same model. There are no human graders yet.
• Workday-hosted postings are not fetched, and the sample still leans toward tech and startup employers.
• AI and ML engineers have no O*NET occupation of their own, so that lane borrows the software developer code.
• The score is not a hiring prediction.

What I’d change in v2

• Add human graders to the consistency eval and report agreement with them.
• Record the model version used for each eval round, so a model upgrade can be compared against it.
• Run the eval in CI when the grading rules change.
• Bring the README up to date. It names three skills and six ship.

Walk me through it

1. Why a deterministic script around a non-deterministic grader still gives a noisy score.
2. How the ±2 target was derived from the size of a single recommendation.
3. The fixtures and the traps planted in them.
4. Round 1: what the model inferred and why that is dangerous for a job seeker.
5. The rules that came out of it, and the round where one lane got worse.
6. What the eval still cannot tell me.

Decision records and related work

• DR-008 and DR-009
• Pull request #36: measure grading consistency and tighten the rules
• Pull request #16: add the résumé market scorer
• AI-Era Job Kit, which builds the full scoring chain on this approach