Asking an AI to do a task is table stakes. The maximization layer is automation that runs without you: hooks that enforce your rules on every action, loops that keep working until a condition is met, and scheduled agents that wake up on a calendar. This Setup File publishes the pieces behind a real one: the pipeline that keeps our LLM Ranking Tool current in a market that changes weekly.
The situation
Our LLM tool ranks current AI models. "Current" is the hard part: in the last ten days alone, GPT-5.6 went GA, Grok 4.5 launched, and Kimi K3 dropped a 2.8 trillion parameter surprise. Hand-updating a dataset like that lasts about two weeks before it quietly rots, and a stale tool is worse than no tool.
So the dataset maintains itself on a schedule, with three layers doing three different jobs. Understanding which layer does what is the entire skill:
- Hooks are reflexes. They fire automatically on events (before or after the AI edits a file, runs a command, or finishes) and they are enforced by the harness, not by the AI remembering.
- Loops are persistence. The AI repeats a task on an interval or until a condition holds, surviving across turns.
- Schedules are calendars. A fresh agent wakes at a set time with a standing brief and does the job with nobody watching.
The files
Layer one: hooks. In Claude Code these live in .claude/settings.json and run shell commands on tool events. Here is a hook pair that guards this very site. The first lints any PHP file the moment the AI edits it; the second enforces a house style rule you will meet again in Setup File 03:
{
"hooks": {
"PostToolUse": [
{
"matcher": "Edit|Write",
"hooks": [{
"type": "command",
"command": "jq -r '.tool_input.file_path // empty' | xargs -r php -l"
}]
},
{
"matcher": "Edit|Write",
"hooks": [{
"type": "command",
"command": "jq -r '.tool_input.file_path // empty' | xargs -r grep -l '—' && echo 'WARN: em dash in copy; house style forbids it'"
}]
}
]
}
}
One mechanical detail worth knowing because every real hook uses it: the hook command receives the event as JSON on standard input, so jq pulls out the edited file's path before the check runs. Beyond that, the point of a hook is that it cannot be forgotten. The AI does not need to remember your linting policy, because the policy runs itself on every edit and feeds the result straight back into the session.
Layer three: the schedule. Claude Code calls these routines: cloud agents that wake on a cron schedule with a standing brief and do the job unattended. A routine is just that brief plus a calendar. Ours, condensed:
Runs: Mondays 06:00 CT
1. Fetch current model catalogs and pricing pages for every provider
in llm-models.json, plus artificialanalysis.ai leaderboards.
2. Diff against the dataset. For anything new, changed, or retired,
gather verified figures with a source URL each. Prefer independent
numbers over vendor numbers; label which is which in "notes".
3. Update the JSON: models, as_of, updated timestamp, and a changelog
entry describing exactly what changed and what was verified.
4. Validate: JSON parses, every model has price and context values,
the page's embedded fallback stays in sync, php -l passes.
5. If nothing changed, still stamp as_of and log "verified, no
changes". An empty week must look different from a broken week.
Step 5 is the one nobody writes and everybody needs. An automation that produces no artifact when idle is indistinguishable from an automation that died in March.
The run
Proof beats promises. This is the actual changelog our pipeline wrote inside llm-models.json across the last week, verbatim:
"as_of": "2026-07-17",
"changelog": [
"Added Kimi K3 (Moonshot, 2026-07-16): 2.8T MoE, 1M context,
AA Intelligence Index 57.",
"Updated Kimi K2.6, Grok 4.5, and Gemini 3.5 Flash notes;
confirmed Gemini 3.5 Pro remains unreleased.",
"Verified against official pricing pages, Artificial Analysis,
and vendor announcements on 2026-07-17."
]
# The previous run, 2026-07-14, three days earlier:
"Expanded the hand-curated set with MiMo-V2.5-Pro, Command A+,
and Mistral Small 4."
"Replaced Mistral Large 3 with the newer Mistral Medium 3.5
flagship."
Every line names what changed and when it was verified. When a client asks "how current is this tool," the answer is a timestamp the pipeline wrote itself, not a shrug.
One more loop pattern worth showing because everyone gets it wrong: waiting on slow work. The wasteful version polls every minute. The right version arms a long fallback timer and lets completion notifications do the waking:
# Two agents dispatched in the background. Instead of polling:
> schedule a fallback check in 25 minutes in case
notifications never arrive
Wakeup armed: 25 min ("waiting on research agents; long
fallback in case completion notifications never fire")
[6 min later] Agent notifications arrive; work continues
immediately. The fallback later fires, confirms everything
is already integrated, and cancels itself.
Same idea in Codex. All three layers exist there under different names. Hooks live in a hooks.json with events that mirror Claude Code's (PreToolUse, PostToolUse, Stop). Schedules are "scheduled tasks": you describe the job and the recurrence is stored as a calendar rule. And the unattended review layer is built in: an automatic-reviews toggle has Codex review every new GitHub pull request, honoring review guidelines it reads from the repo's AGENTS.md. The design rules in this article are tool-agnostic: same contracts, same proof artifacts, same kill switch.
Where it breaks
- No proof artifact. The silent failure is the killer. If the job cannot show a stamp for "I ran and found nothing," you will discover the outage months later inside bad data.
- Drift without contracts. A loop that "keeps things updated" reinterprets its mission a little every run. Ours pins a JSON schema and a validation step; the loop can drift in wording, never in shape.
- Polling too fast. Checking a slow thing every minute burns tokens to learn nothing. Match the check interval to how fast the world actually changes, and prefer notifications with a long fallback over polling at all.
- No kill switch. Every recurring automation needs a documented off command and an owner. An automation nobody can stop is an incident with a calendar.
- Hooks that block instead of inform. A hook that hard-fails on a style nit will stall real work. Ours lint hard (syntax errors block) and style-warn soft. Reserve blocking for things that must never ship.
Steal this
The five-question loop design test
Before automating anything with AI, write one line for each. If you cannot, the automation is not ready:
- Trigger: what starts it (event, interval, calendar)?
- Contract: what exact shape must the output take?
- Proof: what artifact shows it ran, including on a no-change run?
- Validator: what mechanical check runs on the result before anyone trusts it?
- Kill switch: how does a human stop it, and who is that human?
The business translation writes itself: the same three layers keep a price list current, chase stale invoices on a schedule, or lint every outgoing proposal against your house rules. If you have a "someone updates it by hand until they forget" process, that is exactly the shape we automate.
