I Let AI Agents Run a Nationwide CTF. The Hard Part Wasn't Solving the Challenges.

On this page

The official result was 26th place with 5,701 points. I competed individually, and AI use was allowed.

What I wanted to test was not simply whether an LLM could solve CTF challenges. I wanted to see what happened when I treated the whole event as a system: ingest many challenges, distribute them across agents, collect artifacts, verify candidate answers, and submit under a fixed deadline.

The surprising part was that challenge solving itself was not the hardest problem. The harder part was keeping multiple workers aligned on who owned what, what state was authoritative, which files belonged to which challenge, and when a submission should stop.

What I wanted to test

A CTF with many independent challenges looks like a good fit for parallel agents. In principle, the workflow is straightforward: collect the challenge, inspect any attached artifact, assign a worker, generate a candidate answer, verify it, submit it, and move on.

In practice, every one of those arrows needs shared state.

That was the experiment I cared about. This is therefore an operations retrospective, not a challenge-solution write-up. I will keep the official result separate from my local tracker and use the surviving logs only for claims they can actually support.

Result

The local tracker contains 110 challenges. Submission results were recorded for 107 of them: 101 marked correct and 6 marked incorrect. Three rows have no recorded submit result.

Those 107 rows are not an automation count. They only tell me what the tracker retained.

If I add the nominal point values of the 101 rows marked correct, the local total is 5,900 points. The official score was 5,701. I still do not have enough evidence to explain the 199-point gap, so I treat the two numbers as separate records rather than trying to force them to reconcile.

How the run worked

The retained local data includes all 110 challenge records and 21 attachments linked to 20 challenge IDs. Session logs show both local advisor/model activity and remote API use. There are also explicit instructions to split work in parallel and records of a separate submission worker.

So the broad shape of the pipeline is clear: multiple workers were solving in parallel while submission was handled as a separate concern.

What is missing is equally important. I did not preserve a complete per-challenge trace of worker assignment, model routing, human intervention, and retry history. The outcome of a challenge is often visible; the full path that produced it is not.

One of the screenshots taken during the event shows the account at 29th place with 4,401 points. The original capture also contained the OMP pane and challenge-level history, which I did not publish. An even earlier capture, at about 13:53 JST, showed 302nd place with 1 point. That was simply an early event state, not the final result.

I also reconstructed a supplementary timeline from the control-plane events and human-intervention records that survived.

The timeline is useful for seeing what overlapped and when. It is not a stopwatch for challenge solving; I simply did not have that level of per-challenge tracing in the system.

The harder problem was knowing the state

The tracker told me a lot about results, but much less about the live system.

I could often see that a challenge had eventually been solved or submitted. What was much harder to answer was: who owns it right now? Is another worker already solving the same thing? Has a submission result been reflected back into the queue? Should this task still be running at all?

That distinction matters because an HTTP request completing successfully is not the same as the CTF accepting the answer. If workers are operating from copied or stale queue state, or if a submission result arrives late, duplicate work becomes easy. Close to the deadline, stale state can also leave workers running after their work has stopped being useful.

I cannot turn the surviving logs into a reliable count of how often those failure modes actually happened. But the architecture made them possible, and that alone was enough to identify what needed to change.

The bottleneck was the control plane

Going in, I expected model quality to dominate the result. After the event, I cared much more about the control plane.

• Ownership / state: each challenge needs one authoritative owner and an expiring lease. Without that, duplicate work is hard to prevent or even measure.
• Artifact intake: challenge ID, file hash, file type, and provenance should be recorded together so the solver is working on the artifact the controller thinks it is.
• Verification: candidate answers need a separate check for both format and meaning before they reach the submitter.
• Observability: challenge-keyed events should trace ingest → assign → solve → verify → submit → accepted/rejected. Without that trace, model latency and orchestration delay blur together.
• Close control: the official cutoff should stop new work, drain or cancel in-flight tasks, and trigger final result reconciliation.

None of those are glamorous model improvements. They are ordinary distributed-systems problems, which is exactly why they became the limiting factor once several agents were running at the same time.

Local models and APIs

The run definitely used both local models and remote APIs. What I did not record well enough was the routing decision for each challenge.

That means I cannot honestly say that local compute saved a specific number of API calls, or that a specific number of challenges required human intervention. The useful lesson is simpler: route choice itself should be part of the event log. Next time I want to know not only what answer was produced, but which execution path produced it and why.

What I would change next

The next version starts with one canonical queue keyed by challenge ID.

A worker claims a challenge through a lease instead of copying state into its own world. Artifact intake records hashes and provenance. Deterministic local analysis runs first, followed by a health-checked local model or a remote API when needed and within an explicit budget. Candidate answers stay in a private evidence store. An independent verifier gates a single submission broker. At the official cutoff, the broker closes, in-flight work is drained or cancelled, and the final event result is reconciled against the local tracker.

1. 01
   Canonical challenge queueactive lease keyed by challenge ID
2. 02
   Validated artifact intakefile types, hashes, and provenance
3. 03
   Deterministic local analysisfirst proposed processing step
4. 04
   Health-checked model routinglocal model or budgeted remote API
5. 05
   Private evidence and independent verificationgate candidate material before submission
6. 06
   Single submission broker and cutoffclose submissions before score reconciliation

The comparison below separates what the retained records support from what the next design proposes.

Observed

V1 — evidence recovered

Incomplete evidence boundary; not a verified end-to-end pipeline.

• Challenge snapshots110 challenge snapshots
• Local attachments21 files across 20 challenge IDs; linkage is unknown
• Status tracker110 rows; 107 recorded results
• Recorded outcomes101 correct; 6 incorrect

Not recovered from local evidence

• Ingest and assignment method; per-challenge owner; solver linkage
• Submission automation and retries; route telemetry; cutoff reconciliation

Parallel OMP/model activity is observed in traces. Challenge-level assignment and route mapping are not established.

Proposed

V2 — proposed next design

A design proposal, not a description of the event-time implementation.

1. 01
   Challenge sourcePrompts and artifact references
2. 02
   Ingest + artifact integrityHashes, types, and immutable packet
3. 03
   Canonical challenge queue + keyed owner leaseOne owner per challenge; append-only state transitions
4. 04
   Risk-tiered solver routing

   • Local fast
   • Local strong
   • API escalation
5. 05
   Candidate + evidencePrivate; payload in vault
6. 06
   Independent verifierEvidence and format gate
7. 07
   Submission brokerOne request + stop gate
8. 08
   CTFd response + reconciliationHTTP status is not a semantic outcome

Cross-cutting controls (proposed)

• Append-only event ledger
• Per-challenge owner lease
• Human intervention log
• Route/token/cost telemetry
• Global cutoff / close barrier
• Official-state reconciliation

Every item in this panel is a proposal; it is not recovered event-time evidence.

Looking back

The main lesson was not that more agents solve more challenges.

Once the individual workers are capable enough, the limiting problem shifts upward: ownership, state, verification, observability, and shutdown. Adding another worker before fixing those layers can make the system harder to understand rather than faster.

If I run this again, I will instrument those boundaries first. Only after I can answer “who owns this challenge, what happened to it, and why?” will I add more parallelism.

Evidence and scope

• Figure 1 records the final official result: 26th place and 5,701 points.
• The local tracker contains 110 challenges and 107 recorded results: 101 correct, 6 incorrect, and 3 without a recorded result. The nominal local total of the correct rows is 5,900 points; the 199-point difference from the official score remains unexplained.
• Figure 4 is a public crop of an event-time capture at 29th place / 4,401 points. The associated OMP pane and challenge-level history were withheld. The 302nd-place / 1-point capture at about 13:53 JST was an earlier state.
• The supplementary timeline and the V1 side of Figure 5 are reconstructions from retained evidence. V2 is a proposal.
• Per-challenge model routing and human-intervention counts are incomplete, so I do not claim a precise automation percentage.