Modelling what an AI feature will cost before you build it

Conventional software cost is dominated by fixed infrastructure: the same server serves a hundred or ten thousand requests. Hosted language models invert that. Cost tracks usage almost exactly, which makes a cost model a design input rather than a finance exercise afterwards.

Measure tokens, do not estimate them

Every projection starts with tokens per request, measured on your actual evaluation set rather than guessed from a sample prompt. Retrieved context is usually the largest and most underestimated component.

// Instrument from the first prototype. Recording usage per call is the
// only way a projection is grounded rather than assumed.
const response = await model.generate({ messages, maxTokens: 800 });

await recordUsage({
  feature: "document_qa",
  tenantId,
  model: response.model,
  promptTokens: response.usage.promptTokens,
  outputTokens: response.usage.outputTokens,
  retrievedChunks: context.length,
  cacheHit: response.cacheHit ?? false,
  latencyMs: response.latencyMs,
});

// Then the projection is a query, not a spreadsheet guess:
// median and p95 tokens per request, by feature, over real traffic.

The levers, in order of effect

Cost reduction options are not equal. These are ordered by how much they typically save relative to the effort and the quality risk involved.

1. 01Retrieve less context. Passing ten chunks where four suffice inflates every single request. Measure quality against chunk count on your evaluation set — the curve usually flattens earlier than expected.
2. 02Use a smaller model where evaluation permits. Capability differences are task-specific; many extraction and classification tasks show no measurable quality loss on a cheaper model.
3. 03Cache aggressively. Identical or near-identical requests are more common than teams assume, particularly for document questions and summaries.
4. 04Route by difficulty. Send straightforward requests to a cheap model and escalate only the hard ones, using a cheap classifier to decide.
5. 05Cap output length. Verbose answers cost more and are frequently worse; a max-token limit is both a cost and a quality control.
6. 06Trim the system prompt. Lower impact per token, but it is paid on every request forever.

Caching, and what is actually cacheable

There are three distinct caching strategies here and they are frequently conflated. Each addresses a different repetition pattern.

Cache keys must include the tenant, the retrieved context version and the prompt version. A cache that survives a prompt change serves output from a system you no longer run.

Building the projection

With measured tokens, the arithmetic is straightforward and worth writing down explicitly so assumptions are visible and challengeable.

Per request (measured on the evaluation set):
  input tokens  (median / p95)
  output tokens (median / p95)

Per request cost = (input  × input rate)
                 + (output × output rate)

Monthly cost = per-request cost
             × requests per active user per month
             × active users
             × (1 − cache hit rate)

Then state the assumptions separately, because they are what will
turn out to be wrong:
  - requests per user per month
  - cache hit rate
  - whether p95 or median is the realistic operating point
  - provider rates, which change

Run it at three volumes: current, ten times current, and a hundred times. A feature that is comfortable at all three is safe to ship. One that breaks at ten times needs a cost design decision now, not after growth arrives.

Guardrails, because projections are wrong

No model survives contact with real usage patterns. What protects the budget is enforcement at runtime rather than accuracy in the spreadsheet.

• Per-tenant rate limits on AI features specifically, separate from general API limits
• A monthly token budget per tenant, with defined behaviour on exhaustion — degrade or queue rather than fail opaquely
• Alerting on cost per active user, which catches a change in usage pattern that a total-spend alert misses
• A hard provider-side spend cap, so a runaway loop cannot produce an unbounded bill
• Per-feature attribution, so you can tell which feature moved the number

Where this reaches the commercial model

A feature with genuine marginal cost per request sits awkwardly inside unlimited-usage pricing. That is a commercial decision rather than an engineering one, but engineering owns the number the decision depends on.

Producing a defensible cost-per-active-user figure early lets the pricing conversation happen while the product is still flexible — rather than after launch, when the options are raising prices or absorbing the cost.