PODJUDGE

The Ratel Works Podoutside charternot audited by a counterpart

Owner: Glen Warren

What this artifact can and cannot show. Delivered as a signed freeze bundle. The zip's SHA-256 matches the checksum the pod published alongside it, and the repository arrived as a git bundle carrying full history — 926 commits, 7 authors, tagged v1.0.0-freeze. This is the only submission whose integrity is verifiable against a value the pod declared in advance. The sponsor has ruled it delivered on time. It sits outside the signed charter, so no counterpart pod ran its build in the audit exchange; that is disclosed rather than adjusted for.
A Go modular monolith behind one public API (47 paths, 64 operations), built by open-weight crew models under Claude orchestration, delivered as the only submission with full git history (926 commits), a pre-declared verifiable zip hash, and nine reachable staging sites. Its central compliance claims held under attack: the frozen in-process prover was executed for this review and passed C, D, E and F end to end, and no dispatch path bypasses the named-human approval gate. Against that stand three confirmed defects at the freeze tag — a clean-database boot failure introduced by a post-deadline merge, a stale acceptance harness that fails at section A, and no runtime path that captures a live inbound STOP SMS — plus five commits after the 08:00 UTC freeze, two of them new content. This pod entered outside the signed charter; that difference is disclosed, not scored. Because it is the only entrant whose history can be examined at all, the freeze findings here are a product of its transparency, not a penalty for it.
82 / 100 scored

Tokens and spend, as the pod reported them

Tokens reported13,620,898,737
By memberclaude-fable-5 4311M, claude-opus-5 3176M, ar-glm-5.3 1930M, claude-opus-4-8 1702M
Spend reportednot reported
Models namedAnthropic models for orchestration, architecture and review (platform layer under rule 2); ar-glm-5.3, k3, glm5.2 and ar-kimi-k3 on the crew lanes for implementation

Source: repo/docs/submission/token-usage.md — generated from primary sources (Claude transcript JSONL and the OpenCode session store), regenerable with token-report.py. These are the pod's own figures, not measurements by this assessment. Cap: €1,500 per pod.

Effort — stated against demonstrable

Hours statednot reported
Commits926 across 1 repo(s), 7 authors
Active days7 days, 2026-08-20 to 2026-08-31 · busiest 2026-08-27 (442)
Commits inside the 3-hour window33.2%

Full history from a git bundle: 926 commits across 7 authors. Commit timing is verifiable; commit count is not effort.

No hours claimed, so nothing to reconcile. The history shows a concentrated build: the commit profile below is the only evidence of when the work happened.

A commit is not an hour and a commit count is not effort. This bounds when work happened, not how long it took, and no score is derived from it.

Prompt craft 8 / 10

Few prompts, but the best-crafted ones in the trial. The four briefs in docs/submission/prompts/ are prompts-as-work-orders with a specificity level no other pod approaches: GLM-BRIEF cites defects by file and line ('internal/assistant/followup.go:105 tells a paying customer... Nothing sends'), orders work explicitly, and demands a merged/parked/deleted verdict table per branch, which is rule 6 built into the prompt itself; SPRINT-BRIEF embeds measured evidence ('126-197ms means NO MODEL IS INVOLVED in turns 1-5') so the model argues with data, not vibes. The journal closes the loop the other pods leave open: it records corrections to its own earlier entries, an idle-burn cost incident, and a merge-gate that 'reverted eight times', each with the guard that was changed afterwards, so kept-vs-redone and honesty are best-in-class. What holds it at 8: the crew's per-turn prompt stream is not in the submission (AI-USAGE counts it but the transcripts live on the crew box), so day-to-day prompt iteration below brief granularity is asserted rather than shown, and the brief material clusters in the final week of the window.

Assessed separately from the twelve categories: the trial's question was how each pod worked with models, and the prompts are the record of that.

Flagged for a second look

Things a reader would want to know. Not findings against the submission and not scored — questions, oddities and loose ends.

Repositoryratelworks-pod-v1.0.0-freeze.zip via deck-ngaige.staging.con5ult.com/handover/
Received2026-08-31 12:48 CEST

Assessment by category

Acceptance coverage

solid7/9
Do items A–F actually work, on demand, without the pod driving?
Judge justification

C, D, E and F were proven by executing the pod's own frozen prover during this review (gate shut pre-approval and open post-approval, 60 sandbox MSISDNs with HMAC-verified receipt ingest, byte-identical reconciled report, STOP recorded mid-run with the re-dispatch suppressing and a re-import not resurrecting), and the API trace confirms exactly one gateway send call site behind the approval gate. But 'on demand, without the pod driving' failed twice as shipped: the freeze tree cannot boot on a clean database (migration 000062 references a table created only at slot 445901 — reproduced here, SQLSTATE 42P01), and scripts/proof-acceptance.sh fails at section [A] because it still posts to the removed unauthenticated /v1/clients route; F's enforcement is proven but no runtime code path routes a live inbound STOP (SMPP non-receipt deliver_sm is acked and dropped at internal/gateway/smpp.go:876-884, and suppression.RecordStop has no callers outside proof.go and seeddemo.go). B was not independently verifiable live.

What would make this better

Wire non-receipt deliver_sm MOs through suppression.RecordStop, fix the 000062/445901 migration ordering, and update proof-acceptance.sh section [A] to the authenticated /api/v1/clients route so the whole harness passes from a clean clone.

What this category looks for
  • A onboard SME: business name, sender ID, consented recipient list
  • B campaign created conversationally through the Telegram POC
  • C engine drafts the plan and a human must approve before dispatch
  • D dispatch to 50+ test MSISDNs, delivery receipts ingested
  • E report of sent / delivered / failed / cost, reproducible by script
  • F STOP honoured, opted-out excluded from the NEXT send

C and F are refusal properties — the system must decline to do something. A demo can look perfect with the mechanism absent, so these are probed adversarially rather than asked about.

Architecture & technology

solid7/8
Are the technology choices ones you would defend in six months?
Judge justification

The load-bearing choices are defensible: one public contract with no privileged back channel (verified — the only send path is POST /api/v1/dispatch-requests and it refuses unapproved campaigns), ports-and-adapters with a single gateway call site, fail-closed defaults on consent, compliance and suppression checks, and a sovereign LLM router addressed by role names rather than model names. The weakness is schema discipline — burned and renumbered migration slots culminating in the 000062-before-445901 ordering bug that breaks fresh installs — and a 47-path surface far wider than the six acceptance items needed, which is where the pod's own self-reported 5xx breakage class lived.

What would make this better

Add a CI job that boots the freeze tree against an empty PostgreSQL so migration-ordering regressions are caught the day they are merged.

What this category looks for
  • Component boundaries match the problem, not a framework's defaults
  • Dependencies justified; nothing heavy pulled in for a few lines
  • The orchestrator / engine split is real, not cosmetic
  • Failure paths considered: what happens when the gateway is down
  • Choices explained somewhere, not just made

Novelty is not a virtue here. Boring and correct beats clever and unexplained.

Code quality

partial6/8
Could a new engineer change this safely next week?
Judge justification

Discipline is real: spec tests pin behaviour per package, dangerous operations have single call sites, secrets resolve by file path only, and an env-registry drift test exists (and honestly fails at the freeze). But 79,148 lines written in a dense in-house idiom of narrative 'laws' ('the flush law', 'the stopView escape law') raises the cost for a new engineer considerably, a .scratch/ directory of foreign work-in-progress is committed at the root, and the migration renumbering the pod itself admits to twice burned slots.

What would make this better

Write a short glossary of the recurring 'law' vocabulary in CONTRIBUTING.md so a new engineer can decode the comment idiom before their first change.

What this category looks for
  • Readable without a guide; naming that matches the domain
  • No large copy-paste blocks or dead speculative abstraction
  • Errors handled where they occur, not swallowed
  • Consistent style across the codebase, including AI-written parts

Volume earns nothing. A small codebase that does the six things beats a large one that does four.

Testing & verification

solid7/8
Do the tests prove the acceptance items, or just exercise code?
Judge justification

The tests prove acceptance semantics, not just code: TestSpec_D3_ApprovalGateHonoured, D6_StopSuppressedSkipped, D10b_StopBeatsConsent and the frozen ngaige proof subcommand map directly onto C, D, E and F, and the full suite reproduced here exactly as documented (71 packages ok, one disclosed failure). Yet the pod's own admission stands confirmed — three simultaneous staging 5xx were found by hand-probing, the outer acceptance harness rotted against a route change without any test noticing, and the migration ordering bug shipped despite a dedicated migrations test package.

What would make this better

Add one integration test that runs scripts/proof-acceptance.sh (or its section A equivalent) against a spawned API in CI so the harness can never drift from the routes again.

What this category looks for
  • Tests exist, run from a clean clone, and pass
  • The refusal properties (C, F) have tests of their own
  • Verification evidence beyond unit tests: scripts, fixtures, logs
  • A failing test would actually catch a real regression

Coverage percentage is not read. A single test that proves STOP excludes on the next send is worth more than 80% line coverage.

Deployability & operations

partial5/8
Clean clone to running system, following only the written steps.
Judge justification

The written deploy story is the best of the three submissions — an idempotent one-command installer with --check/--verify/--rollback/--uninstall, a doctor subcommand, an env registry with a drift test, and an agent-readable DEPLOY-PROMPT.md. But the category asks for clean clone to running by the written steps, and that was executed here and failed: boot migration 000062 (landed via the 08:56 post-freeze merge) errors with SQLSTATE 42P01 on a fresh database, and the API only came up after manually pre-creating the contact_attributes table.

What would make this better

Rehearse install.sh on a genuinely empty host from the tagged tree before every freeze — the last recorded clean-host rehearsal (2026-08-27) predates the migration that broke it.

What this category looks for
  • One documented command, or a short documented sequence
  • Configuration and secrets handled sanely; .env.example present
  • It actually starts — verified, not asserted
  • Teardown, logs and health checks exist

This is scored by doing it. The result is binary and unkind: either the written steps produced a running system or they did not.

Documentation & handover

solid8/8
Could HAUD take this over without the pod in the room?
Judge justification

HAUD could take this over without the pod in the room: deploy README plus runbooks, threat model, engineer onboarding, a decision register, a security register with open items, five generated SDKs, and a developer portal whose client guide reproduces executed wire transcripts. Spot-checks of cited claims were almost all exact (79,148/109,853 line counts, 47 paths/64 operations, the named failing test); the misses are small — 'prover log' is claimed in docs/submission/evidence/ but only demo-readiness.txt is there, and '124 migrations' counts up and down files of 62 migration pairs.

What would make this better

Ship the prover output log the submission cites, or amend section 3 to say only the readiness snapshot was copied in.

What this category looks for
  • The submission .md answers all seven requested points
  • Claims cite where in the repo the implementation lives
  • Each claim says how to reproduce or test it
  • A runbook for operating it, not just a README for reading it

The bar is the client-bound mainline: would you hand this to someone who has never met the authors?

AI usage & prompt craft

solid8/9
The A/B payload — how the pod actually worked with models.
Judge justification

The strongest A/B payload of the three entrants: per-model token tables counted from primary transcripts with a committed regenerator, real working lane briefs (GLM-BRIEF.md is an actual order-of-work reconciling 28 stranded branches), a documented orchestration/implementation split, and self-measured failure modes including the 3-minute keepalive dependency and the 97% cache-read ratio. Held back from top marks by a scope inconsistency between its own two measurements: token-report.py counts every transcript under ~/.claude/projects with no date filter, so the headline 13.62B includes pre-run work that AI-USAGE.md explicitly excludes from its 4.96B figure.

What would make this better

Add the 2026-08-20 scope cut to token-report.py so the committed report and AI-USAGE.md measure the same estate.

What this category looks for
  • Prompt library or logs present and legible
  • Which models, used where, and why those
  • Evidence of iteration: what was kept, what was redone
  • Token usage per model where recorded
  • Human review of model output visible in the history

This is the whole point of the trial. A pod that shipped well but cannot show how it used AI has answered the wrong question.

Process & decision evidence

partial6/8
Decision log and daily journal — kept as work, not written on the 30th.
Judge justification

The history is manifestly kept-as-work — 926 commits, a timestamped event journal with costed incidents and owner rulings, decision records and gap analyses that long predate the freeze. Rule 4 is where it breaks: freeze was 08:00 UTC and five commits land after it — the 08:26 and 08:56 merges integrate work authored 00:57-07:57 (integration late, content not), but f12c372 (08:14) is a single-parent commit with new conflict-resolution changes, 90c742d (10:02) is 36 lines of new product code two hours past freeze, and e7f459f (10:30) assembles the submission documents into the repo and takes the tag; the 08:56 merge also introduced the clean-install regression. This is the only pod whose freeze could be examined at all, precisely because it supplied full history — the scrutiny is a consequence of its superior provenance, not a sanction for it.

What would make this better

Tag the tree that existed at the deadline and deliver post-deadline assembly as a separate clearly-labelled commit or branch, so the freeze tag itself is unarguable.

What this category looks for
  • Decisions recorded with their reasoning and their date
  • Journal entries across the working days, not backfilled in a burst
  • Impediments logged with category, owner and time lost
  • Rework recorded honestly, including what was thrown away

Entries dated across fifteen days read differently from fifteen entries committed on the last afternoon. The git history shows which.

Candour & self-assessment

solid8/9
The requested LLM assessment, and whether it tells the truth.
Judge justification

The self-assessment tells the truth and was verified doing so: the named failing test reproduces exactly, section 5 leads with the worst items (no load test, no independent review, tests missing real breakage, poor token economics), the security register keeps findings open rather than declaring them fixed, and both falsifiable claims held when attacked — no send path bypasses approval, and Vouch's marketing consent is a separate off-by-default switch. Two blemishes: SUBMISSION.md presents the freeze without disclosing that the tag and two content commits post-date the deadline, and the 13.62B headline is mis-scoped against the pod's own earlier measurement.

What would make this better

Add the post-deadline commit timeline to the submission document — a submission this candid should not leave its one awkward fact for the reviewer to discover.

What this category looks for
  • The LLM-generated assessment is present
  • It names areas where evidence is weak or the requirement is unmet
  • Known issues, limitations and shortcuts are listed specifically
  • The self-assessment matches what the repo and probes actually show

A submission that names its own gaps is more trustworthy than one that claims six of six. Overclaiming found by probe is scored here, and it is the most damaging single finding a submission can carry.

Model choice & sovereignty

solid7/8
Which models, were they the right ones, and did they respect the boundary?
Judge justification

Implementation ran on open-weight models (GLM 5.2/5.3, Kimi K3) on rented GPU instances behind the pod's own LiteLLM router, and the product at runtime speaks only to that router by role name over private addresses (deploy/app.env.example; internal/agent/llm) — no third-party AI API appears in any product path. Anthropic models did architecture, review, verification, the demo apps outside this repo and the submission document itself, which is a far broader role than the charter's 'platform author with no repo access' contemplates — but this pod entered outside the charter and its AI-USAGE.md discloses the division plainly.

What would make this better

State explicitly in AI-USAGE.md whether any Anthropic model wrote or edited files inside the product repo, since two Glen-authored commits (90c742d, e7f459f) were assembled within the Claude estate.

What this category looks for
  • Models named specifically, with versions — not 'an LLM'
  • Product code and trial data touched self-hosted open weights only
  • Third-party APIs, if used, stayed on the platform layer (rule 2)
  • Model choice fits the task: a frontier model where it earned its cost, a small one where it did not
  • The issued rig was actually used, not idled while work went elsewhere

Newest is not best. The scored question is fit and boundary, not leaderboard position. A pod that ran a mid-tier open model well scores above one that reached for a frontier API it was not permitted to use. A rule-2 breach is a finding here regardless of how good the output was.

Token & spend efficiency

partial6/9
What the budget bought — EUR 1,500 per pod, hard, every euro logged.
Judge justification

Token accounting is exemplary and self-critical (the 97% cache-read waste is stated, not netted out), and the journal logs burn continuously — hourly rates, balance snapshots, top-ups, and a costed idle-burn incident. But the consolidated euro ledger the journal repeatedly cites (LEDGER.md) is not in the freeze, the ~EUR 225 idle-burn was excluded from the EUR 1,500 cap by the pod's own ruling rather than an external one, and the security register mentions a EUR 1,773 unbudgeted-key spend on a shared account whose cap treatment is nowhere reconciled — so 'every euro logged' cannot be verified from the submission alone.

What would make this better

Commit LEDGER.md to the repo with a final total against the EUR 1,500 cap, showing the carve-out and the EUR 1,773 shared-account incident explicitly.

What this category looks for
  • Token usage recorded per model, not guessed at the end
  • Spend logged as it happened, with the cap respected
  • Output per euro: what the money actually produced
  • Rented GPU hours matched to work — parked when idle, not left burning
  • Rework cost visible: what was regenerated and why

Underspending is not virtue and overspending is not vice; an exhausted budget is a workflow finding, not a failure. What is scored is whether the pod knew where the money went and can show it. A pod that spent the full cap and can account for it scores above one that spent half and cannot.

Security & data handling

solid7/8
Consent, opt-out, secrets and the sovereignty boundary.
Judge justification

Verified posture is strong: every API route returns 401 unauthenticated including dispatch (probed live), no plaintext secrets in the repo, secrets by file path only, key records digest-only after a disclosed plaintext-mint finding, DLR authenticity HMAC-verified before parse, tenant-scoped suppression that fails closed on store errors, and an honest register of what remains open (SMPP TLS via stunnel workaround, staging/production pepper). The real gap is compliance-critical: a live inbound STOP over the carrier link never reaches the suppression store because non-receipt deliver_sm PDUs are dropped after ack, so the opt-out capture half of F exists only on the sandbox/prover path.

What would make this better

Route non-receipt deliver_sm bodies through suppression.RecordStop (and add a Twilio inbound-message webhook doing the same) with a spec test asserting a wire STOP suppresses the next send.

What this category looks for
  • Consent recorded on the recipient list, not assumed
  • Opt-out state durable and honoured across campaigns
  • No credentials committed; secrets loaded from environment
  • Trial data synthetic; sovereignty boundary respected
  • Sender ID and MSISDN handling sane

A campaign platform that leaks a recipient list or ignores an opt-out is not a product, whatever else it does well.

Evidence

CategoryItemWhereNote
acceptanceFrozen A-F prover executed for this review: PROOF PASS/home/erian/pod-judge/submissions/pod3-glen/repo/cmd/ngaige/proof.goRun against a scratch PostgreSQL: SECTION [C] gate_closed_pre_approval=true gate_open_post_approval=true; [D] 60 sent, 60 receipts applied; [F] redispatch_suppressed=1 tombstone=true resurrected=false; [E] reconciles_D=true. Required a manual workaround for the migration ordering bug first.
deployabilityClean-database boot failure at the freeze tag/home/erian/pod-judge/submissions/pod3-glen/repo/migrations/000062_contact_attr_value_enc.up.sqlReproduced: 'relation "contact_attributes" does not exist (SQLSTATE 42P01)' — the table is created only in 445901_contacts_rich.up.sql, which runs after 000062. 000062 was authored 07:57 on a lane branch and landed via the 08:56 post-freeze merge (0328181).
acceptanceAcceptance harness fails at section [A] on a removed route/home/erian/pod-judge/submissions/pod3-glen/repo/scripts/proof-acceptance.shHarness posts to /v1/clients; cmd/ngaige/main.go:1544 prints 'legacy unauthenticated onboarding mount REMOVED (brief 0.2 — use /api/v1/clients with a scoped key)'. Reproduced: PROOF-ACCEPTANCE FAIL '[A] create client status 404'.
securityLive inbound STOP is dropped on the SMPP path/home/erian/pod-judge/submissions/pod3-glen/repo/internal/gateway/smpp.goLines 871-884: a deliver_sm is acked, and only IsReceipt PDUs are forwarded; a non-receipt MO (a subscriber texting STOP) goes nowhere. suppression.Service.RecordStop has no product-runtime callers — only cmd/ngaige/proof.go and seeddemo.go.
acceptanceThe approval gate is structural/home/erian/pod-judge/submissions/pod3-glen/repo/internal/dispatch/service.goLine 337: Dispatch calls gate.ReleaseForDispatch first; internal/campaign/service.go requires StateApproved plus a stored approval row, and Approve() (line 661) refuses without a named approver and a liability acknowledgement. gateway.Send has exactly one product call site (dispatch/service.go:551).
processPost-freeze commits (freeze 08:00 UTC)/home/erian/pod-judge/submissions/pod3-glen/repof12c372 08:14 (single-parent 'merge', 34 insertions of new conflict resolution), 527ae8b 08:26 and 0328181 08:56 (true merges of work authored 00:57-07:57), 90c742d 10:02 (36 new lines in internal/assistant/service.go), e7f459f 10:30 (submission assembly, 30 files, 3,987 insertions — the tagged commit).
candourDisclosed failing test reproduces exactly/tmp/claude-1000/-home-erian-pod-judge/90dd2927-2f25-457a-a631-b3b42b914dce/scratchpad/gotest.loggo test ./... on the freeze tree: 71 packages ok, sole failure TestRegistryCoversEveryEnvTheCodeReads in cmd/ngaige — precisely as SUBMISSION.md section 5.8 and README state.
documentationCited line and API counts verified exact/home/erian/pod-judge/submissions/pod3-glen/docs/SUBMISSION.md79,148 product / 109,853 test lines and 47 paths / 64 operations all reproduce to the digit with the commands the document supplies. Misses: 'prover log' absent from docs/submission/evidence/, and '124 migrations' counts up+down files of 62 pairs.
ai_usageToken report scope inconsistency/home/erian/pod-judge/submissions/pod3-glen/repo/docs/submission/token-report.pyCounts all ~/.claude/projects transcripts with no date filter (13.62B headline), while decisions/AI-USAGE.md scopes from 2026-08-20 and excludes an earlier project (4.96B measured 08-29).
securityAPI surface locked, livehttps://developer-ngaige.staging.con5ult.com/api/v1/dispatch-requestsProbed: POST unauthenticated and with the site basic-auth both return 401 problem+json; POST /api/v1/keys likewise. OpenAPI carries no send operation outside the campaign lifecycle.
candourConsent-unbundled claim supported on Vouchhttps://vouch.staging.con5ult.com/Review form renders marketing consent as a separate switch, aria-checked=false by default, outside the primary action ('the consent moment — unbundled, off by default').
efficiencySpend narrated in the journal, ledger not shipped/home/erian/pod-judge/submissions/pod3-glen/repo/docs/submission/decisions/journal.mdPer-event balances, hourly rates ($60/hr K3 pod), top-ups, a costed ~EUR 240 idle-burn with an owner-ruled EUR 225 carve-out from the EUR 1,500 cap; LEDGER.md is cited but absent from the freeze. SECURITY-REGISTER.md S8 records a EUR 1,773 unbudgeted-key spend on a shared account.
processProvenance verified end to end/home/erian/pod-judge/submissions/pod3-glen/_bare/ratelworks-pod-v1.0.0-freeze.zipZip SHA-256 matches the pre-declared 5b5533a2...869f; bundle and tarball match SHA256SUMS. The only submission whose integrity is checkable against a value the pod published in advance.

Questions for the pod

Raised by the assessment, not answered by it. Each is a thing a reader of the submission would reasonably ask.

Could not verify

Checks that could not run. Recorded as unverified, never counted as failures.