Cheap tokens do not guarantee cheap outcomes. Divide the cost of all observed attempts by tasks that actually succeeded, using one reporting window and a success definition you can audit.
Calculate with your assumptions
Sample inputs are illustrative. Calculations run locally. Input values are not sent to analytics or included in a shared link; only tool event names and the tool ID are measured.
Cost per successful task, not cost per API call
Total observed cost = all model attempts × average cost per attempt + external tool cost
Cost per success = total observed cost / successfully completed tasks
The average must include successful, failed, retried and cancelled calls. External cost is the total for the same window, not a per-call rate; exclude any item already included in the model cost. At least one model attempt per successful task is assumed. Use task IDs to deduplicate completions.
Example: why the cheapest model can lose
150 attempts at $0.02 each plus $1 of tool cost totals $4. With 80 completed tasks, cost per success is $0.05. If a lower-priced setup costs $3 but completes only 40 tasks, it costs $0.075 per success. The first setup is more expensive per period but cheaper per completed outcome; neither comparison proves equal answer quality.
Handle zero success and missing usage honestly
Zero completed tasks means cost per success is undefined, not $0. If even one billed attempt has missing usage or pricing, uncheck complete evidence: total cost and cost per success stay unknown. Do not divide the cost of only successful calls by all tasks.
Define success before comparing providers: for example, a correctly reconciled invoice accepted by a reviewer. Keep rejected, timed-out and duplicate tasks out of the success count. Compare identical task mixes and windows, not one easy demo against a production workload.
Use it to improve an agent
- Join every attempt to an opaque task ID and record the observed result.
- Reconcile usage and prices, including failed requests and external tools.
- Change one factor, such as cache reuse or retry policy; compare both success rate and cost per success.
- Check contribution margin after payment fees and refunds; lower operating cost is not automatically profit.
Continue with the retry waste calculator, prompt cache calculator, and AI contribution margin calculator.
Methodology and evidence boundary
This is a transparent arithmetic tool, not a benchmark or model recommendation. Inputs are user estimates until reconciled with time-versioned provider prices and observed request usage. No production customer data or model call is required.