Benchmark report
Task verified
9m 21s elapsed
The saved attempt met the documented success criterion.
81/ 100
Needs Alignment- SInstall
- SAuth
- AExecute
- BDocs
- ACurrent
Chasing a Clay-managed routine id that the API documents but exposes no way to list, forcing a fallback to the Searches endpoints for the stripe.com lookup.
Where the time went
9:21 agent time- Research0:28
- Attempt6:49
- Verify2:03
Judge reason
Independently verified from saved artifacts and transcript tool output, not prose. Clay's own openapi.json (saved) documents GET /credits/balance ('Reading balances does not consume credits') and POST /search/query-mode + /search/query-mode/{search_id}/run. code/reconcile_credits.sh ran live: run_log.txt and summary.txt show HTTP 200 for balance_before, search_create, search_results, balance_after; 02b_search_results.json returns exactly one real company record (Stripe, stripe.com, linkedin.com/company/stripe, period_quota used 578993) so nothing is mocked. Balance before and after are both 226157.4999999997, difference 0.0, with a T+90s recheck (04_balance_delayed_check.json, HTTP 200) ruling out delayed accounting. The report correctly states the docs publish no per-lookup credit price: grep of the saved llms-full.txt confirms 'credit' appears only on the balance page, so the agent cited the gap instead of inventing a price, and explained the zero delta via the response's separate record period_quota. The routines path was not guessed away: probes of plausible Clay-managed ids returned 404/400 (routine_id_probes.txt) and the saved docs show routine ids are 'function:t_...' values copied from the app UI or MCP list_subroutines, with no list endpoint in openapi.json. No saved file, log or transcript line contains a token; the API uses a clay-api-key header read from a private file via a temp header file, and an independent grep for bearer/access_token/long token strings across /workspace/home found nothing.
Attempt record
- 1:24Answered from saved key: Clay personal API key (clay-api-key header)no wait
- —First success verified
Attempt artifacts saved.