Benchmark methodology (v0)

Weekly, Mondays 06:00 UTC, GitHub Actions, US-East runners, 10 runs per provider, default or cheapest tier. We publish p50/p90, failures, and raw JSON. The harness is open source.

Sandboxes: cold start → run → teardown

1. Create a fresh sandbox via the official SDK.
2. Run: pip install requests; sha256 over 50 MB of generated bytes; write and read a 100 MB file.
3. Destroy.

Metrics: create→ready ms, pip ms, CPU ms, IO ms, total ms, success rate, estimated $/run.

Browsers: open → navigate → extract → screenshot

1. Connect via CDP (Playwright connectOverCDP).
2. Load a static fixture page, then news.ycombinator.com.
3. Extract 30 titles, full-page screenshot, close.

Metrics: session create ms, navigation ms, extraction correctness, success rate, estimated $/session.

Sponsors cannot change the method or the results.