PODJUDGE

Pod Trials — independent findings

Ten categories, ten points each, scored against what the submitted repository, the submission document and a live probe actually show. Every score carries a written justification and a note on what would make it better.

This is not the panel's result. The Pod Trials charter locks a separate six-axis rubric scored by a three-person panel on the day. This assessment is independent of it and decides nothing. How this was assessed.
The Pod of Conviction
Luke
81 / 100
The Ratel Works Podoutside charter
Glen Warren
82 / 100
The Prove-It Pod
71 / 100

Category comparison

CategoryPod of ConvictionRatel Works PodProve-It PodMax
Acceptance coverage
Do items A–F actually work, on demand, without the pod driving?
6749
Architecture & technology
Are the technology choices ones you would defend in six months?
7768
Code quality
Could a new engineer change this safely next week?
6668
Testing & verification
Do the tests prove the acceptance items, or just exercise code?
6758
Deployability & operations
Clean clone to running system, following only the written steps.
7568
Documentation & handover
Could HAUD take this over without the pod in the room?
7878
AI usage & prompt craft
The A/B payload — how the pod actually worked with models.
8879
Process & decision evidence
Decision log and daily journal — kept as work, not written on the 30th.
7678
Candour & self-assessment
The requested LLM assessment, and whether it tells the truth.
7879
Model choice & sovereignty
Which models, were they the right ones, and did they respect the boundary?
8778
Token & spend efficiency
What the budget bought — EUR 1,500 per pod, hard, every euro logged.
8679
Security & data handling
Consent, opt-out, secrets and the sovereignty boundary.
4728
Total818271100

Tokens and spend

PodTokens reportedSpend reportedCap
The Pod of Conviction1,009,989,029$1,446 ≈ €1,242€1,500
The Ratel Works Pod13,620,898,737not reported€1,500
The Prove-It Pod1,114,540,355$1,158€1,500

Each pod's own figures, from its own records — not measured by this assessment, and not comparable like for like: the pods counted different things. Per-pod pages give the source and the caveats.

Read the column, not the ranking. One pod counts cache reads, which dominate an agent workload and inflate a total by roughly an order of magnitude against a pod counting input and output alone. One reports GLM-5.2 only. One excludes its DevOps AI usage by its own note. These are three different measurements wearing the same unit.

Effort — stated against demonstrable

PodHours statedCommitsActive daysIn the 3-h window
The Pod of Convictionnot reported4581421.0%
The Ratel Works Podnot reported926733.2%
The Prove-It Pod299 hno history

Charter §2: pod work is confined to a shared three-hour window per working day — 09:00-12:00 CEST — with automation excepted and humans not. 'The cap is the experiment's control: up to forty-five pod-hours per person, identical for both pods by construction.' Work outside the window was to be visible, and time lost to business-as-usual logged as an impediment.

One pod reported hours; two did not. The pod that reported them is the only one whose figures can be questioned — which is a consequence of disclosure, not of working longer. A commit is not an hour, and no score is derived from any number in this table.

Prompt craft

PodScoreAssessment
The Pod of Conviction5 / 10The strongest individual prompt documents sit next to the weakest evidence trail. The in-repo artifacts are genuinely good: prompt.md and APP_GENERATION_PROMPT.md carry hard constraints (exact stack, 'Do not us…
The Ratel Works Pod8 / 10Few prompts, but the best-crafted ones in the trial. The four briefs in docs/submission/prompts/ are prompts-as-work-orders with a specificity level no other pod approaches: GLM-BRIEF cites defects by file and …
The Prove-It Pod8 / 10The most complete prompt record of the trial, and most of it is verbatim. Five workstreams each left artifacts in their own working style: the FE series is day-numbered (Day 1-12) with real API schemas, polling…

Scored separately from the twelve categories. The trial's question was how each pod worked with models, and the prompts are the record of that. Volume was not rewarded — the largest library did not score highest.

Patterns across the pods

The categories

Acceptance coverage 9 pts

Do items A–F actually work, on demand, without the pod driving?
  • A onboard SME: business name, sender ID, consented recipient list
  • B campaign created conversationally through the Telegram POC
  • C engine drafts the plan and a human must approve before dispatch
  • D dispatch to 50+ test MSISDNs, delivery receipts ingested
  • E report of sent / delivered / failed / cost, reproducible by script
  • F STOP honoured, opted-out excluded from the NEXT send

Architecture & technology 8 pts

Are the technology choices ones you would defend in six months?
  • Component boundaries match the problem, not a framework's defaults
  • Dependencies justified; nothing heavy pulled in for a few lines
  • The orchestrator / engine split is real, not cosmetic
  • Failure paths considered: what happens when the gateway is down
  • Choices explained somewhere, not just made

Code quality 8 pts

Could a new engineer change this safely next week?
  • Readable without a guide; naming that matches the domain
  • No large copy-paste blocks or dead speculative abstraction
  • Errors handled where they occur, not swallowed
  • Consistent style across the codebase, including AI-written parts

Testing & verification 8 pts

Do the tests prove the acceptance items, or just exercise code?
  • Tests exist, run from a clean clone, and pass
  • The refusal properties (C, F) have tests of their own
  • Verification evidence beyond unit tests: scripts, fixtures, logs
  • A failing test would actually catch a real regression

Deployability & operations 8 pts

Clean clone to running system, following only the written steps.
  • One documented command, or a short documented sequence
  • Configuration and secrets handled sanely; .env.example present
  • It actually starts — verified, not asserted
  • Teardown, logs and health checks exist

Documentation & handover 8 pts

Could HAUD take this over without the pod in the room?
  • The submission .md answers all seven requested points
  • Claims cite where in the repo the implementation lives
  • Each claim says how to reproduce or test it
  • A runbook for operating it, not just a README for reading it

AI usage & prompt craft 9 pts

The A/B payload — how the pod actually worked with models.
  • Prompt library or logs present and legible
  • Which models, used where, and why those
  • Evidence of iteration: what was kept, what was redone
  • Token usage per model where recorded
  • Human review of model output visible in the history

Process & decision evidence 8 pts

Decision log and daily journal — kept as work, not written on the 30th.
  • Decisions recorded with their reasoning and their date
  • Journal entries across the working days, not backfilled in a burst
  • Impediments logged with category, owner and time lost
  • Rework recorded honestly, including what was thrown away

Candour & self-assessment 9 pts

The requested LLM assessment, and whether it tells the truth.
  • The LLM-generated assessment is present
  • It names areas where evidence is weak or the requirement is unmet
  • Known issues, limitations and shortcuts are listed specifically
  • The self-assessment matches what the repo and probes actually show

Model choice & sovereignty 8 pts

Which models, were they the right ones, and did they respect the boundary?
  • Models named specifically, with versions — not 'an LLM'
  • Product code and trial data touched self-hosted open weights only
  • Third-party APIs, if used, stayed on the platform layer (rule 2)
  • Model choice fits the task: a frontier model where it earned its cost, a small one where it did not
  • The issued rig was actually used, not idled while work went elsewhere

Token & spend efficiency 9 pts

What the budget bought — EUR 1,500 per pod, hard, every euro logged.
  • Token usage recorded per model, not guessed at the end
  • Spend logged as it happened, with the cap respected
  • Output per euro: what the money actually produced
  • Rented GPU hours matched to work — parked when idle, not left burning
  • Rework cost visible: what was regenerated and why

Security & data handling 8 pts

Consent, opt-out, secrets and the sovereignty boundary.
  • Consent recorded on the recipient list, not assumed
  • Opt-out state durable and honoured across campaigns
  • No credentials committed; secrets loaded from environment
  • Trial data synthetic; sovereignty boundary respected
  • Sender ID and MSISDN handling sane