Adversarial round: a webhook service
A harder second test: build a payment-webhook service, then attack it with 15 scenarios, for zerd and Claude Code alike.
A second, harder test than the 11 tasks: build an inbound payment-webhook service (signature check, idempotency, an outbox, queue workers, retries, admin replay), then attack it. Same 15 scenarios for both agents: 100 identical webhooks at once, two events for one payment, a worker killed mid-job, Redis down, Postgres down, retries running out, replay, bad signatures, malformed and oversized bodies, unauthorised and cross-tenant admin access, out-of-order events, a crash between commit and ack.
| zerd | Claude Code | |
|---|---|---|
| Scenarios with an automated test | 15 / 15 | 14 / 15 (one kept as a known failure, test.fail()) |
| Database | real PostgreSQL (a schema-only copy) | in-memory repository |
| Queue | real Redis client, outages simulated by pointing it at dead ports | enqueue made to throw |
| Worker crash | a real child process killed with SIGKILL | lease expiry simulated |
| HTTP | the real Fastify app via inject | service layer only |
| Concurrency | 25 repeated races on one payment | Promise.all of 100 in memory |
| Bugs found | 12, each reproduced or checked directly | 3, each reproduced (one left open) |
| Left open | tenant trust, shared secrets, public metrics | out-of-order events |
What each found:
- zerd (infrastructure): unquoted camelCase SQL so no webhook ever persisted; a lost update that left a captured payment
FAILED; retries scheduled about 5.5 hours late on an IST machine; Redis down making enqueue hang so the outbox stopped draining; an event stuck inRECEIVEDafter a database outage; two dispatchers enqueuing the same event; a crash after the effect marking an eventIGNORED; BullMQ rejecting job ids that contain:. Fixes went in one commit each (21 commits). - Claude Code (service logic): an
undefinedexception that crashed the failure path, admin retries corrupting the attempt numbering, and, the one zerd did not find, a refund applying before its capture when events arrive out of order.
Read it honestly:
- The two runs were not under the same conditions. Claude Code tested the service layer only, so this compares depth of testing as
much as quality of engineering. Neither side is ready for production: zerd's own report lists merchant-controlled tenant ids, one
shared Razorpay secret, one shared admin token and unauthenticated
/metrics. - zerd changed one assertion (a signed JSON array now expects
invalid_payload, notinvalid_signature) and said so. It also flagged that one commit message overstates what it verified. - No winner is declared. The follow-up is a controlled round: Claude Code re-run against real Postgres, Redis, worker processes and SIGKILL, and zerd given a security and architecture audit of its own result. Results go here when they are in.