How to Benchmark an AI Coding Gateway Reproducibly

Observed results — October 16, 2026

Both clients ran sequentially on the same macOS machine against https://api.superfastcode.com, in empty temporary directories, with the fixed prompt “Do not use tools or inspect any files. Reply with exactly: superfastcode-benchmark-ok”. No warm-up calls were excluded and no project files or tools were used.

First text delta was measured by a local HTTP observer from starting HTTPS forwarding to receiving the first nonempty protocol text delta. It includes network/TLS time and observer overhead; it is not gateway-only latency. CLI elapsed includes startup and final-result validation. HTTP 200, a completed protocol stream, exit status zero and the exact final marker were required for success.

Codex used low reasoning effort, ephemeral execution and no user config; ambient client/global instruction metadata still produced about 20.6k input tokens including 4,096 cached tokens per request. Claude used bare mode, no tools/MCP/skills, low effort and zero thinking budget, with 139–141 input tokens per request. Different models and contexts prevent a direct latency or cost comparison. Client location and gateway/upstream regions were not independently verified.

A read-only check matched all ten billing records by narrow UTC time windows, model and exact token counts. Recorded deductions were 0.00167850 for Codex and 0.00002148 for Claude, in USD-denominated account-credit units. Both test credentials had an observed 0.01 billing multiplier. These are test-account deductions, not public customer prices, cash payments or provider invoices. Use current public pricing and your actual account terms for an estimate.

Download all ten sanitized observations and the measurement definitions (JSON). The test window was 07:06–07:10 UTC. Five samples in one short window cannot establish p95, an SLA, uptime or coding quality. No comparative route was tested.

Control the comparison before collecting numbers

Use the same client and version, exact model, reasoning setting, prompt, output requirement, machine, region, network and concurrency. Record the UTC test window and route configuration without publishing credentials. Changing the model or reasoning level changes the experiment.

Test one route at a time using its legitimate client configuration. Alternate routes across comparable time windows instead of running all requests to one first and all requests to another much later. If access rules require different clients or models, label those as separate experiments rather than claiming a gateway-only speed difference.

Use multiple tasks for real work: a small patch with tests, a constrained refactor, and a repository question with verified source references. Run them in disposable test projects. The tiny marker prompt below tests basic completion only, not code quality.

Download and run the client harness

Download gateway-benchmark.py. It uses Python’s standard library, starts the installed CLI without a shell, records one JSONL row per attempt and prints a summary. Configure the client using its setup guide first. Each attempt can send a paid model request; select a small count for your first run.

python3 gateway-benchmark.py --client codex --model gpt-6.1-sol --route-label superfastcode --runs 5 --output codex-results.jsonl

python3 gateway-benchmark.py --client claude --model claude-sonnet-5-5 --route-label superfastcode --runs 5 --output claude-results.jsonl

The tool refuses to overwrite an existing result file, caps the requested run count, spaces requests and applies a per-run timeout. It records model, client version, route label, timestamps, exit status, marker success and elapsed milliseconds. It does not save raw prompts, model outputs or credentials. User-level client settings and hooks remain active; review them or use a clean test profile. Keep client-managed logs and output files private.

Expected output: one progress line per attempt followed by attempts, completed marker responses, success rate, median and p95 elapsed time for successful attempts. Failures remain in the denominator. A timeout or failed command is not silently excluded.

Know what each metric actually measures

For a more stable latency distribution, collect at least 30 attempts per condition across multiple periods when your budget permits. Declare warm-up policy, cache state, retry behavior and concurrency in advance. Do not delete slow runs simply because they look inconvenient.

Publish enough evidence to reproduce a result

A useful report includes the date, exact client version, requested and observed model, endpoint region, task fixtures, command, context and reasoning settings, attempt count, success criteria, all failure categories, latency definition and actual billing method. State which fields were not observed instead of guessing.

Redact account IDs, credentials, private code and provider-specific secrets before sharing. Publish a small public fixture and sanitized results only when authorized. A repeatable method and raw evidence are more useful than an unexplained “fastest” badge.

Sources checked October 16, 2026: Official OpenAI non-interactive CLI documentation; Claude Code CLI reference.