Owner: Glen Warren
| Tokens reported | 13,620,898,737 |
|---|---|
| By member | claude-fable-5 4311M, claude-opus-5 3176M, ar-glm-5.3 1930M, claude-opus-4-8 1702M |
| Spend reported | not reported |
| Models named | Anthropic models for orchestration, architecture and review (platform layer under rule 2); ar-glm-5.3, k3, glm5.2 and ar-kimi-k3 on the crew lanes for implementation |
Source: repo/docs/submission/token-usage.md — generated from primary sources (Claude transcript JSONL and the OpenCode session store), regenerable with token-report.py. These are the pod's own figures, not measurements by this assessment. Cap: €1,500 per pod.
| Hours stated | not reported |
|---|---|
| Commits | 926 across 1 repo(s), 7 authors |
| Active days | 7 days, 2026-08-20 to 2026-08-31 · busiest 2026-08-27 (442) |
| Commits inside the 3-hour window | 33.2% |
Full history from a git bundle: 926 commits across 7 authors. Commit timing is verifiable; commit count is not effort.
A commit is not an hour and a commit count is not effort. This bounds when work happened, not how long it took, and no score is derived from it.
Few prompts, but the best-crafted ones in the trial. The four briefs in docs/submission/prompts/ are prompts-as-work-orders with a specificity level no other pod approaches: GLM-BRIEF cites defects by file and line ('internal/assistant/followup.go:105 tells a paying customer... Nothing sends'), orders work explicitly, and demands a merged/parked/deleted verdict table per branch, which is rule 6 built into the prompt itself; SPRINT-BRIEF embeds measured evidence ('126-197ms means NO MODEL IS INVOLVED in turns 1-5') so the model argues with data, not vibes. The journal closes the loop the other pods leave open: it records corrections to its own earlier entries, an idle-burn cost incident, and a merge-gate that 'reverted eight times', each with the guard that was changed afterwards, so kept-vs-redone and honesty are best-in-class. What holds it at 8: the crew's per-turn prompt stream is not in the submission (AI-USAGE counts it but the transcripts live on the crew box), so day-to-day prompt iteration below brief granularity is asserted rather than shown, and the brief material clusters in the final week of the window.
submissions/pod3-glen/repo/docs/submission/prompts/GLM-BRIEF.mdsubmissions/pod3-glen/repo/docs/submission/prompts/SPRINT-BRIEF.mdsubmissions/pod3-glen/repo/docs/submission/decisions/AI-USAGE.mdAssessed separately from the twelve categories: the trial's question was how each pod worked with models, and the prompts are the record of that.
Things a reader would want to know. Not findings against the submission and not scored — questions, oddities and loose ends.
submissions/pod3-glen/repo/docs/submission/decisions/journal.mdsubmissions/pod3-glen/repo (git log --date=short)submissions/pod3-glen/repo/docs/submission/decisions/AI-USAGE.mdsubmissions/pod3-glen/repo/docs/submission/decisions/journal.mdsubmissions/pod3-glen/repo/docs/submission/decisions/journal.mdsubmissions/pod3-glen/repo/docs/submission/decisions/journal.md| Repository | ratelworks-pod-v1.0.0-freeze.zip via deck-ngaige.staging.con5ult.com/handover/ |
|---|---|
| Received | 2026-08-31 12:48 CEST |
C, D, E and F were proven by executing the pod's own frozen prover during this review (gate shut pre-approval and open post-approval, 60 sandbox MSISDNs with HMAC-verified receipt ingest, byte-identical reconciled report, STOP recorded mid-run with the re-dispatch suppressing and a re-import not resurrecting), and the API trace confirms exactly one gateway send call site behind the approval gate. But 'on demand, without the pod driving' failed twice as shipped: the freeze tree cannot boot on a clean database (migration 000062 references a table created only at slot 445901 — reproduced here, SQLSTATE 42P01), and scripts/proof-acceptance.sh fails at section [A] because it still posts to the removed unauthenticated /v1/clients route; F's enforcement is proven but no runtime code path routes a live inbound STOP (SMPP non-receipt deliver_sm is acked and dropped at internal/gateway/smpp.go:876-884, and suppression.RecordStop has no callers outside proof.go and seeddemo.go). B was not independently verifiable live.
Wire non-receipt deliver_sm MOs through suppression.RecordStop, fix the 000062/445901 migration ordering, and update proof-acceptance.sh section [A] to the authenticated /api/v1/clients route so the whole harness passes from a clean clone.
C and F are refusal properties — the system must decline to do something. A demo can look perfect with the mechanism absent, so these are probed adversarially rather than asked about.
The load-bearing choices are defensible: one public contract with no privileged back channel (verified — the only send path is POST /api/v1/dispatch-requests and it refuses unapproved campaigns), ports-and-adapters with a single gateway call site, fail-closed defaults on consent, compliance and suppression checks, and a sovereign LLM router addressed by role names rather than model names. The weakness is schema discipline — burned and renumbered migration slots culminating in the 000062-before-445901 ordering bug that breaks fresh installs — and a 47-path surface far wider than the six acceptance items needed, which is where the pod's own self-reported 5xx breakage class lived.
Add a CI job that boots the freeze tree against an empty PostgreSQL so migration-ordering regressions are caught the day they are merged.
Novelty is not a virtue here. Boring and correct beats clever and unexplained.
Discipline is real: spec tests pin behaviour per package, dangerous operations have single call sites, secrets resolve by file path only, and an env-registry drift test exists (and honestly fails at the freeze). But 79,148 lines written in a dense in-house idiom of narrative 'laws' ('the flush law', 'the stopView escape law') raises the cost for a new engineer considerably, a .scratch/ directory of foreign work-in-progress is committed at the root, and the migration renumbering the pod itself admits to twice burned slots.
Write a short glossary of the recurring 'law' vocabulary in CONTRIBUTING.md so a new engineer can decode the comment idiom before their first change.
Volume earns nothing. A small codebase that does the six things beats a large one that does four.
The tests prove acceptance semantics, not just code: TestSpec_D3_ApprovalGateHonoured, D6_StopSuppressedSkipped, D10b_StopBeatsConsent and the frozen ngaige proof subcommand map directly onto C, D, E and F, and the full suite reproduced here exactly as documented (71 packages ok, one disclosed failure). Yet the pod's own admission stands confirmed — three simultaneous staging 5xx were found by hand-probing, the outer acceptance harness rotted against a route change without any test noticing, and the migration ordering bug shipped despite a dedicated migrations test package.
Add one integration test that runs scripts/proof-acceptance.sh (or its section A equivalent) against a spawned API in CI so the harness can never drift from the routes again.
Coverage percentage is not read. A single test that proves STOP excludes on the next send is worth more than 80% line coverage.
The written deploy story is the best of the three submissions — an idempotent one-command installer with --check/--verify/--rollback/--uninstall, a doctor subcommand, an env registry with a drift test, and an agent-readable DEPLOY-PROMPT.md. But the category asks for clean clone to running by the written steps, and that was executed here and failed: boot migration 000062 (landed via the 08:56 post-freeze merge) errors with SQLSTATE 42P01 on a fresh database, and the API only came up after manually pre-creating the contact_attributes table.
Rehearse install.sh on a genuinely empty host from the tagged tree before every freeze — the last recorded clean-host rehearsal (2026-08-27) predates the migration that broke it.
This is scored by doing it. The result is binary and unkind: either the written steps produced a running system or they did not.
HAUD could take this over without the pod in the room: deploy README plus runbooks, threat model, engineer onboarding, a decision register, a security register with open items, five generated SDKs, and a developer portal whose client guide reproduces executed wire transcripts. Spot-checks of cited claims were almost all exact (79,148/109,853 line counts, 47 paths/64 operations, the named failing test); the misses are small — 'prover log' is claimed in docs/submission/evidence/ but only demo-readiness.txt is there, and '124 migrations' counts up and down files of 62 migration pairs.
Ship the prover output log the submission cites, or amend section 3 to say only the readiness snapshot was copied in.
The bar is the client-bound mainline: would you hand this to someone who has never met the authors?
The strongest A/B payload of the three entrants: per-model token tables counted from primary transcripts with a committed regenerator, real working lane briefs (GLM-BRIEF.md is an actual order-of-work reconciling 28 stranded branches), a documented orchestration/implementation split, and self-measured failure modes including the 3-minute keepalive dependency and the 97% cache-read ratio. Held back from top marks by a scope inconsistency between its own two measurements: token-report.py counts every transcript under ~/.claude/projects with no date filter, so the headline 13.62B includes pre-run work that AI-USAGE.md explicitly excludes from its 4.96B figure.
Add the 2026-08-20 scope cut to token-report.py so the committed report and AI-USAGE.md measure the same estate.
This is the whole point of the trial. A pod that shipped well but cannot show how it used AI has answered the wrong question.
The history is manifestly kept-as-work — 926 commits, a timestamped event journal with costed incidents and owner rulings, decision records and gap analyses that long predate the freeze. Rule 4 is where it breaks: freeze was 08:00 UTC and five commits land after it — the 08:26 and 08:56 merges integrate work authored 00:57-07:57 (integration late, content not), but f12c372 (08:14) is a single-parent commit with new conflict-resolution changes, 90c742d (10:02) is 36 lines of new product code two hours past freeze, and e7f459f (10:30) assembles the submission documents into the repo and takes the tag; the 08:56 merge also introduced the clean-install regression. This is the only pod whose freeze could be examined at all, precisely because it supplied full history — the scrutiny is a consequence of its superior provenance, not a sanction for it.
Tag the tree that existed at the deadline and deliver post-deadline assembly as a separate clearly-labelled commit or branch, so the freeze tag itself is unarguable.
Entries dated across fifteen days read differently from fifteen entries committed on the last afternoon. The git history shows which.
The self-assessment tells the truth and was verified doing so: the named failing test reproduces exactly, section 5 leads with the worst items (no load test, no independent review, tests missing real breakage, poor token economics), the security register keeps findings open rather than declaring them fixed, and both falsifiable claims held when attacked — no send path bypasses approval, and Vouch's marketing consent is a separate off-by-default switch. Two blemishes: SUBMISSION.md presents the freeze without disclosing that the tag and two content commits post-date the deadline, and the 13.62B headline is mis-scoped against the pod's own earlier measurement.
Add the post-deadline commit timeline to the submission document — a submission this candid should not leave its one awkward fact for the reviewer to discover.
A submission that names its own gaps is more trustworthy than one that claims six of six. Overclaiming found by probe is scored here, and it is the most damaging single finding a submission can carry.
Implementation ran on open-weight models (GLM 5.2/5.3, Kimi K3) on rented GPU instances behind the pod's own LiteLLM router, and the product at runtime speaks only to that router by role name over private addresses (deploy/app.env.example; internal/agent/llm) — no third-party AI API appears in any product path. Anthropic models did architecture, review, verification, the demo apps outside this repo and the submission document itself, which is a far broader role than the charter's 'platform author with no repo access' contemplates — but this pod entered outside the charter and its AI-USAGE.md discloses the division plainly.
State explicitly in AI-USAGE.md whether any Anthropic model wrote or edited files inside the product repo, since two Glen-authored commits (90c742d, e7f459f) were assembled within the Claude estate.
Newest is not best. The scored question is fit and boundary, not leaderboard position. A pod that ran a mid-tier open model well scores above one that reached for a frontier API it was not permitted to use. A rule-2 breach is a finding here regardless of how good the output was.
Token accounting is exemplary and self-critical (the 97% cache-read waste is stated, not netted out), and the journal logs burn continuously — hourly rates, balance snapshots, top-ups, and a costed idle-burn incident. But the consolidated euro ledger the journal repeatedly cites (LEDGER.md) is not in the freeze, the ~EUR 225 idle-burn was excluded from the EUR 1,500 cap by the pod's own ruling rather than an external one, and the security register mentions a EUR 1,773 unbudgeted-key spend on a shared account whose cap treatment is nowhere reconciled — so 'every euro logged' cannot be verified from the submission alone.
Commit LEDGER.md to the repo with a final total against the EUR 1,500 cap, showing the carve-out and the EUR 1,773 shared-account incident explicitly.
Underspending is not virtue and overspending is not vice; an exhausted budget is a workflow finding, not a failure. What is scored is whether the pod knew where the money went and can show it. A pod that spent the full cap and can account for it scores above one that spent half and cannot.
Verified posture is strong: every API route returns 401 unauthenticated including dispatch (probed live), no plaintext secrets in the repo, secrets by file path only, key records digest-only after a disclosed plaintext-mint finding, DLR authenticity HMAC-verified before parse, tenant-scoped suppression that fails closed on store errors, and an honest register of what remains open (SMPP TLS via stunnel workaround, staging/production pepper). The real gap is compliance-critical: a live inbound STOP over the carrier link never reaches the suppression store because non-receipt deliver_sm PDUs are dropped after ack, so the opt-out capture half of F exists only on the sandbox/prover path.
Route non-receipt deliver_sm bodies through suppression.RecordStop (and add a Twilio inbound-message webhook doing the same) with a spec test asserting a wire STOP suppresses the next send.
A campaign platform that leaks a recipient list or ignores an opt-out is not a product, whatever else it does well.
| Category | Item | Where | Note |
|---|---|---|---|
| acceptance | Frozen A-F prover executed for this review: PROOF PASS | /home/erian/pod-judge/submissions/pod3-glen/repo/cmd/ngaige/proof.go | Run against a scratch PostgreSQL: SECTION [C] gate_closed_pre_approval=true gate_open_post_approval=true; [D] 60 sent, 60 receipts applied; [F] redispatch_suppressed=1 tombstone=true resurrected=false; [E] reconciles_D=true. Required a manual workaround for the migration ordering bug first. |
| deployability | Clean-database boot failure at the freeze tag | /home/erian/pod-judge/submissions/pod3-glen/repo/migrations/000062_contact_attr_value_enc.up.sql | Reproduced: 'relation "contact_attributes" does not exist (SQLSTATE 42P01)' — the table is created only in 445901_contacts_rich.up.sql, which runs after 000062. 000062 was authored 07:57 on a lane branch and landed via the 08:56 post-freeze merge (0328181). |
| acceptance | Acceptance harness fails at section [A] on a removed route | /home/erian/pod-judge/submissions/pod3-glen/repo/scripts/proof-acceptance.sh | Harness posts to /v1/clients; cmd/ngaige/main.go:1544 prints 'legacy unauthenticated onboarding mount REMOVED (brief 0.2 — use /api/v1/clients with a scoped key)'. Reproduced: PROOF-ACCEPTANCE FAIL '[A] create client status 404'. |
| security | Live inbound STOP is dropped on the SMPP path | /home/erian/pod-judge/submissions/pod3-glen/repo/internal/gateway/smpp.go | Lines 871-884: a deliver_sm is acked, and only IsReceipt PDUs are forwarded; a non-receipt MO (a subscriber texting STOP) goes nowhere. suppression.Service.RecordStop has no product-runtime callers — only cmd/ngaige/proof.go and seeddemo.go. |
| acceptance | The approval gate is structural | /home/erian/pod-judge/submissions/pod3-glen/repo/internal/dispatch/service.go | Line 337: Dispatch calls gate.ReleaseForDispatch first; internal/campaign/service.go requires StateApproved plus a stored approval row, and Approve() (line 661) refuses without a named approver and a liability acknowledgement. gateway.Send has exactly one product call site (dispatch/service.go:551). |
| process | Post-freeze commits (freeze 08:00 UTC) | /home/erian/pod-judge/submissions/pod3-glen/repo | f12c372 08:14 (single-parent 'merge', 34 insertions of new conflict resolution), 527ae8b 08:26 and 0328181 08:56 (true merges of work authored 00:57-07:57), 90c742d 10:02 (36 new lines in internal/assistant/service.go), e7f459f 10:30 (submission assembly, 30 files, 3,987 insertions — the tagged commit). |
| candour | Disclosed failing test reproduces exactly | /tmp/claude-1000/-home-erian-pod-judge/90dd2927-2f25-457a-a631-b3b42b914dce/scratchpad/gotest.log | go test ./... on the freeze tree: 71 packages ok, sole failure TestRegistryCoversEveryEnvTheCodeReads in cmd/ngaige — precisely as SUBMISSION.md section 5.8 and README state. |
| documentation | Cited line and API counts verified exact | /home/erian/pod-judge/submissions/pod3-glen/docs/SUBMISSION.md | 79,148 product / 109,853 test lines and 47 paths / 64 operations all reproduce to the digit with the commands the document supplies. Misses: 'prover log' absent from docs/submission/evidence/, and '124 migrations' counts up+down files of 62 pairs. |
| ai_usage | Token report scope inconsistency | /home/erian/pod-judge/submissions/pod3-glen/repo/docs/submission/token-report.py | Counts all ~/.claude/projects transcripts with no date filter (13.62B headline), while decisions/AI-USAGE.md scopes from 2026-08-20 and excludes an earlier project (4.96B measured 08-29). |
| security | API surface locked, live | https://developer-ngaige.staging.con5ult.com/api/v1/dispatch-requests | Probed: POST unauthenticated and with the site basic-auth both return 401 problem+json; POST /api/v1/keys likewise. OpenAPI carries no send operation outside the campaign lifecycle. |
| candour | Consent-unbundled claim supported on Vouch | https://vouch.staging.con5ult.com/ | Review form renders marketing consent as a separate switch, aria-checked=false by default, outside the primary action ('the consent moment — unbundled, off by default'). |
| efficiency | Spend narrated in the journal, ledger not shipped | /home/erian/pod-judge/submissions/pod3-glen/repo/docs/submission/decisions/journal.md | Per-event balances, hourly rates ($60/hr K3 pod), top-ups, a costed ~EUR 240 idle-burn with an owner-ruled EUR 225 carve-out from the EUR 1,500 cap; LEDGER.md is cited but absent from the freeze. SECURITY-REGISTER.md S8 records a EUR 1,773 unbudgeted-key spend on a shared account. |
| process | Provenance verified end to end | /home/erian/pod-judge/submissions/pod3-glen/_bare/ratelworks-pod-v1.0.0-freeze.zip | Zip SHA-256 matches the pre-declared 5b5533a2...869f; bundle and tarball match SHA256SUMS. The only submission whose integrity is checkable against a value the pod published in advance. |
Raised by the assessment, not answered by it. Each is a thing a reader of the submission would reasonably ask.
Checks that could not run. Recorded as unverified, never counted as failures.