Prompt caching and batch processing: calculate the cost of your actual workload

1. Choose work that fits the mechanism

Caching targets repeated prompt content under the provider’s matching and lifetime rules. Batch processing targets work that can wait for an asynchronous result. A real-time user interaction and an overnight evaluation do not have the same constraints.

Read the current documentation for the exact model and endpoint. Verify minimum eligible input, cache lifetime, retention requirements and supported batch features. A feature available for one model or hosting platform should not be assumed to exist everywhere.

2. Keep a dated rate card

Record provider, model identifier, currency, price unit, effective date and source URL. Separate ordinary input, cached input, cache creation where separately priced, output and other metered tools. Record the applicable batch rates separately.

Use the current rate card for your contract and platform. This guide deliberately does not freeze a model price into a permanent savings claim. Discount rules may differ, and you should not multiply two advertised discounts unless the provider documents that combination.

3. Count mutually exclusive token categories

Start from measured usage for the same set of tasks. If the provider’s total input count already includes cached tokens, subtract the cached subset before charging ordinary input. Otherwise you count the same tokens twice. Apply the same care to cache-write categories and output.

Divide token counts by the rate unit: a price per million tokens requires counts in millions. Add separately billed tools or storage only when applicable. Preserve both the calculated estimate and the provider’s billed amount; their difference may need investigation.

4. Evaluate a representative window

Measure a workload long enough to include warm and cold cache behavior. For batches, count completed, failed, expired and repeated jobs. A larger discount does not help if your deadline requires re-running work through a synchronous endpoint.

• Keep the same acceptance criteria and task mix in both runs.
• Measure cache hits and misses instead of assuming perfect reuse.
• Include retried work and non-token charges.
• Record completion delay, operational effort and accepted outcomes.

5. Decide using verified outcomes

Compare total attributable cost per accepted outcome after billing catches up. A provider-rate saving and a net operational saving are different statements: engineering effort and ongoing maintenance can change the decision.

Keep the simpler workflow if the measured gain does not justify the added complexity. Revisit the calculation when the model, prompt structure, traffic mix or tariff changes. No universal percentage applies to every workload.