Measure AI agent cost per run

Add up every model call in an agent task, compare median and expensive runs, and find retry loops before choosing a per-task price.

By Keyveve, Inc. · Updated 2026-10-23

An agent task can make one model call or hundreds. The price of its final response misses planning, tool-selection steps, retries and other model calls. Measure the whole run before deciding what a completed task costs.

Define the unit of work

Choose a run ID for one user-visible task: a support conversation, a research job or a document-processing request. Attach the same ID to every model call in that task, including retries. Keep customer for the organization paying you and feature for the product capability. Use pseudonymous IDs rather than names, emails or prompt text.

import veqen
from openai import OpenAI

client = veqen.wrap(OpenAI(), feature="support-agent")
with veqen.tags(run="task-123", customer="customer-456"):
    first = client.responses.create(model="gpt-4o-mini", input="Draft a support answer.")
    second = client.responses.create(model="gpt-4o-mini", input="Check this answer: " + first.output_text)
veqen.flush()

This illustrative two-call task incurs provider charges when run. Set VEQEN_API_KEY to a send-only key to send usage to the ledger; without it, the SDK prices locally. The SDK records usage and tags, not the example's prompt or answer. In an asynchronous or serverless app, preserve the tag context and flush at the end of the invocation.

JavaScript uses the same boundary with await withTags({ run: taskId, customer: customerId }, () => runTask()). The OpenAI Agents integration can assign trace IDs automatically. Install one recording path for each call; adding a wrapper and a trace exporter to the same call can double-count it.

Add the costs, then inspect the distribution

Run cost is the sum of its recorded model-call costs. For example, a task with eight $0.002 calls and two $0.02 calls costs $0.056 in model inference. Ten calls at the price of the final $0.002 response would incorrectly suggest $0.02.

Compare three figures over the same time window:

• Total cost divided by completed tasks: useful for the overall inference budget. Track completed tasks in your product so failed or unfinished work is not mistaken for delivered value.
• Median recorded run: what the middle observed run costs.
• Expensive runs: the p90 and largest runs show how a few customers or loops can dominate spend.

Veqen's Agent runs page groups received calls; it does not independently know whether your business task completed. Its percentile fields are approximate. Missing usage, reused run IDs and delayed events affect interpretation. Use a new run ID for a new task and record retries consistently.

Investigate a costly run

Open the run's calls in order. Look for repeated model/tool cycles, a costly model used for a small classification step, or a growing context sent at every step. Compare similar successful tasks before changing the model. A cheaper model that needs more retries can cost more per completed task.

The authenticated MCP tools get_runs and get_run provide this view to an assistant. Ask it to report the date range, number of runs and missing-price limitations before recommending a change. Savings and the cost planner help evaluate model and host choices against recorded usage.

Separate inference from the full cost of a task

The run's model calls do not automatically include search-provider fees, code-execution sandboxes, storage, human review or payment fees. Keep those costs alongside inference when evaluating a product. Compare request-level estimates to provider bills, then connect revenue for customer margin.

Use budgets and your own step, retry and timeout limits together. Post-call tracking cannot stop a request already made, and concurrent calls can exceed a shared observed-spend limit.