An AI teaching assistant that is not allowed to hand over the answer

The problem

Vets Who Code teaches people who are changing careers, often while working another job. They need help at the moment they are stuck, and mentors are volunteers with their own schedules. A general chatbot solves the wrong problem. It gives the answer, and a learner who is handed answers does not pass a technical interview.

What it has to do

• Teach, don’t tell. Solutions are withheld until a learner has taken at least two hints or failed at least three times.
• Stay inside the curriculum. Answers are grounded in approved lessons before anything else.
• Protect learner data. Personal fields are encrypted and every access to them is logged.
• Stay within a nonprofit’s budget. Model spend per learner is capped.
• Degrade without falling over. A failing model must not take the service down.

Approach and architecture

The service is layered: API routes call repositories, repositories own models and schemas, and a core package owns everything that touches a model. Retrieval is a three-tier ladder scoped to the learner’s current module and the two on either side of it. Responses stream to the client over server-sent events.

Data model

PostgreSQL is the system of record. Vector embeddings for the three retrieval tiers live in the same database through pgvector, so retrieval and relational data share one transaction boundary and one backup. Schema changes go through Alembic, and CI runs each migration down and back up. Personal fields are encrypted at the field level with rotating keys, and a dedicated table audits every read of them.

Key tradeoffs

Gate the solution behind effort

Withholding answers frustrates a learner in the moment. It is also the point of the program. The gate is enforced by the solution endpoint itself, so it does not depend on the model choosing to behave. Recorded as DR-011.

The backend records and coaches. It never grades.

Learner code runs in the browser. The server never executes untrusted code, which removes a whole class of security work. The cost is that the backend cannot independently verify a result.

Mock every AI call in tests

More than 670 tests run in about eight seconds because no test calls a model. The suite is fast, free and deterministic, and it proves the plumbing. It does not measure answer quality, which needs a separate eval. Recorded as DR-012.

Hash-pinned lockfile, weekly canary

Production installs from a lockfile with hashes. A scheduled job installs from open version ranges once a week and runs the suite, so an upstream break shows up in the canary instead of during a release.

Measurable outcome

• One engineer built and operates a backend with nine product areas and seven durable workflows.
• CI enforces a 70% coverage floor and runs unit tests, integration tests against a real database, a migration round trip, ruff, mypy, a convention linter, frontend type check, tests and build, and a security scan.
• 10% of successful model responses, and every error and retry, are captured as snapshots for incident replay, with 30-day retention.

Evals, cost, latency, fallback and limits

Prompt injection and output validation

• Input: learner text is sanitized and wrapped in delimiters before it reaches a prompt. The system prompt carries a canary token the model is told never to emit, so a leak is detectable.
• Output: guardrails check each response for tone drift, refusals and personal data before it is returned.
• Data: field-level encryption with key rotation, and an audit log of personal data access.

AI collaboration note

J0DI3 is built with Claude Code, and the repository carries the tooling that keeps that honest: agents that audit a change before a pull request, audit migrations, audit generated content and verify production after a deploy, and skills that scaffold a new endpoint, migration or AI surface the same way each time. I own the architecture, the eight behaviour rules the assistant cannot break, and every merge.

What I’d change in v2

• Build an answer-quality eval with graded examples for each behaviour rule, and run it on every model or prompt change.
• Pin dated model snapshots. The current names follow a model family, so behaviour can shift underneath the tests.
• Publish the p50 and p95 latency and the cost per request that the admin endpoint already reports.

Walk me through it

1. Why a helpful chatbot is the wrong product for someone learning to code.
2. The request path, from learner text to streamed response.
3. The retrieval ladder and why it is scoped to nearby modules.
4. How the solution gate is enforced outside the model.
5. What happens when the model provider fails.
6. What 670 mocked tests prove, and what they cannot.
7. How one engineer keeps this reviewable with AI tooling.

Decision records and related posts

• DR-011 and DR-012
• 300 Veterans Later and a Decade of Teaching Veterans to Code
• Harnessing AI to Transform Coding Education for Veterans