Benchmark report

Task verified

4m 8s elapsed

The saved attempt met the documented success criterion.

Experience score

94/ 100

Agent-Ready
  • AInstall
  • SAuth
  • SExecute
  • ADocs
  • SCurrent
Experience measures friction in the tested task. The verdict says whether it succeeded.
Agent profile for this run: Install A, Auth S, Execute S, Docs A, Current SAInstallSAuthSExecuteADocsSCurrent
Agent profile · this run
Judge's note · what slowed the agent most
Concept docs omitted HTTP methods/paths, forcing an extra hop via llms.txt to the api-reference pages; plus one trivial module-import retry.

Where the time went

4:09 agent time
  • Research0:33
  • Attempt2:22
  • Verify1:14
4m 8sTotal elapsed
1:43Time to first success
0:00Waited on the owner

Judge reason

Live run verified. Saved code (agentmail-task/am_client.py + send_transactional_email.py) makes an authenticated Bearer request to AgentMail's documented send operation POST /v0/inboxes/{inbox_id}/messages/send; the key is read from a private file at runtime, never hard-coded. Evidence 01/02-create-*-inbox.json show POST /v0/inboxes -> 200 creating sender tx-sender-1794253614@agentmail.to and recipient tx-recipient-1794253614@agentmail.to (AgentMail documents no test recipient, so the inbox fallback applies; 00-discover-sender.txt shows GET /v0/inboxes 200 and GET /v0/domains 200 with 0 custom domains, hence the default agentmail.to sender). 03-send-message.json records the live request body {to, subject 'Your AgentMail order #TX-7741 is confirmed', plain-text body} identical to the constants in the code, and response HTTP 200 with message_id <010001a1c1d972c1-...@email.amazonses.com> and thread_id 20a1aae7-b3ba-4a62-94a5-c910da3138d0 — the documented shape. Transcript tool output (run-stdout.txt) prints the same status and ids, so printed values match the saved response; evidence was written programmatically by the executed script, not fabricated. Cleanup evidence 04-delete-inboxes.json shows DELETE /v0/inboxes/{id} -> 202 for both inboxes. No SMTP or dashboard was used.

Attempt record

  1. 1:32Answered from saved key: AGENTMAIL_API_KEYno wait
  2. —First success verified

Attempt artifacts saved.