Prompt optimization is not just about making instructions shorter. It is a controlled way to reduce input and output tokens, improve latency, and keep answer quality within an acceptable range. This guide gives developers a repeatable method for estimating LLM costs, identifying safe prompt compression opportunities, and testing whether a shorter prompt still performs well.
Overview
Every LLM request has a token footprint. The input may include a system prompt, user message, conversation history, retrieved documents, tool definitions, and few-shot examples. The output adds another variable because a model may generate far more text than the application needs. Providers commonly calculate usage by input and output tokens, but the exact pricing, caching treatment, context limits, and tokenizer behavior vary by model and can change over time.
A useful cost estimate therefore starts with a simple equation:
Estimated request cost = (input tokens ÷ 1,000,000 × input rate) + (output tokens ÷ 1,000,000 × output rate)
Use the actual unit rates for the model and account configuration you are evaluating. If a provider uses a different billing unit, such as 1,000 tokens, adapt the denominator. For monthly planning, multiply the per-request estimate by expected requests:
Estimated monthly cost = estimated request cost × requests per month
Prompt optimization should not be judged by token reduction alone. A shorter prompt that causes incorrect tool calls, incomplete answers, or additional retries may cost more overall. Measure at least four outcomes: input tokens, output tokens, response latency, and task quality. For production applications, also track retry rate and the percentage of requests that require human correction.
How to estimate
Begin with a representative request rather than the smallest possible test case. Select a sample that reflects normal inputs, long inputs, edge cases, and any retrieved context. Then separate the request into components:
- Fixed instructions: system guidance, output rules, safety constraints, and role definitions.
- Dynamic instructions: the current user request, account data, conversation state, and tool results.
- Examples: few-shot demonstrations or formatting references.
- Retrieved context: documents, search results, database records, or conversation excerpts.
- Expected output: the answer, structured object, code, classification, or tool call.
Count tokens with the tokenizer associated with the model when possible. Character counts and word counts can help compare drafts, but they are only rough proxies. Code, identifiers, markup, URLs, and text in languages with different segmentation patterns may tokenize differently than ordinary prose.
Create a small worksheet for each prompt version:
| Measure | Baseline | Optimized |
|---|---|---|
| Average input tokens | Record observed value | Record observed value |
| Average output tokens | Record observed value | Record observed value |
| Input rate | Use current provider rate | Use current provider rate |
| Output rate | Use current provider rate | Use current provider rate |
| Requests per month | Use observed or forecast volume | Use the same volume |
| Quality score | Evaluate against a fixed rubric | Evaluate against the same rubric |
For example, suppose a hypothetical model uses an input rate of 2 units per million tokens and an output rate of 8 units per million tokens. A request with 4,000 input tokens and 600 output tokens would be estimated as follows: (4,000 ÷ 1,000,000 × 2) + (600 ÷ 1,000,000 × 8) = 0.008 + 0.0048, or 0.0128 units per request. This is an illustrative calculation, not a current provider price.
At 100,000 similar requests, the estimate would be 1,280 units before considering caching, discounts, failed requests, retries, or other account-specific billing rules. Replace every illustrative input with the values from your own usage and provider documentation.
Inputs and assumptions
Before trimming a prompt, document what must remain true. A practical quality contract might require the model to cite only supplied evidence, return valid JSON, identify uncertainty, preserve all required fields, or call a tool when a specific condition is met. These requirements are more important than preserving every sentence in the original prompt.
Common prompt compression techniques include:
- Remove duplicated instructions. If the same formatting rule appears in the system message, user template, and example, keep the clearest version in one place.
- Replace prose with a compact schema. For stable structured tasks, define required fields, allowed values, and a short example instead of several paragraphs of explanation.
- Reduce example count. Keep examples that represent difficult distinctions or common errors. Remove examples that demonstrate the same pattern.
- Limit conversation history. Summarize completed turns, retain unresolved decisions, and remove greetings or repeated context.
- Filter retrieved context. Retrieve fewer, more relevant passages and remove duplicated headers, navigation text, and irrelevant metadata.
- Constrain output length. Ask for the smallest useful response, specify a field limit, or use a structured format when the task allows it.
- Move stable content out of each request. Depending on provider features, reusable prompt sections may qualify for caching or another optimization. Verify the current implementation and billing treatment before assuming savings.
Do not compress blindly. Removing a security boundary, tool-use condition, or source-handling rule can create quality and reliability problems. For applications handling untrusted content, keep instructions distinct from retrieved text and user-provided data. The Prompt Injection Defense Guide provides additional guidance on separating these elements.
Worked examples
Example 1: Shortening a classification prompt
A baseline prompt includes a 900-token role description, 12 examples totaling 1,800 tokens, a 300-token label definition, and a 200-token user message. Its average input is therefore 3,200 tokens. Testing shows that six carefully selected examples cover the important distinctions, reducing the examples to 850 tokens. A revised role description removes repetition and falls to 350 tokens. The new average input is 1,700 tokens, a reduction of 1,500 tokens per request.
If the output remains a short label, input reduction will usually matter more than output trimming. However, the revised prompt should be tested against a labeled evaluation set. Compare accuracy, confusion between similar labels, invalid labels, and abstention behavior before adopting it.
Example 2: Reducing RAG context
A retrieval-augmented application sends eight passages at 500 tokens each, plus 600 tokens of instructions and 400 tokens of user context. The request contains approximately 5,000 input tokens. A retrieval change sends the four most relevant passages, removes repeated document titles, and reduces instructions to 350 tokens. The revised request contains approximately 2,750 tokens.
The saving is meaningful only if the removed passages were not necessary to answer the question. Test questions that require information from multiple documents, conflicting sources, or a detail near the end of a passage. For grounded applications, score citation support and refusal behavior alongside answer correctness. See the RAG Prompting Guide for patterns that help preserve grounding while changing context.
Example 3: Output control
An assistant is asked to summarize support tickets but often produces 500-token explanations when the application needs three fields: issue, cause, and next action. A structured response contract with field limits can reduce output to an expected 120 tokens. If output pricing is higher than input pricing for the selected model, this may have a disproportionate effect on cost. It can also reduce downstream parsing work and latency.
Do not assume that a shorter maximum output always produces a better result. Set a limit that allows the model to complete the task, then test truncation, missing fields, and unusually complex tickets.
When to recalculate
Recalculate your estimate whenever a key input changes. The most obvious triggers are a provider pricing update, a model change, a new tokenizer, a different context-window configuration, or a change in caching eligibility. Also recalculate when traffic grows, the average user message becomes longer, retrieval returns more content, or product requirements increase the expected output length.
Operational changes can matter even when the prompt has not changed. A new feature may add tool definitions to every request. Longer chat history may increase input tokens. A retry policy may multiply the number of billable requests. A model migration may reduce the token count but alter quality, latency, or output length.
Set a recurring review based on the volatility of your application. High-volume systems benefit from monitoring token usage continuously and reviewing outliers weekly. Smaller applications can review after releases, model changes, and material traffic shifts. Keep baseline and optimized prompt versions in your evaluation suite so a cost improvement can be checked against quality over time.
For a practical optimization cycle, export a representative request sample, calculate baseline input and output tokens, and label the quality requirements. Make one focused change at a time: remove duplication, reduce examples, filter context, or constrain output. Run the same evaluation set, compare quality and latency, then update the cost worksheet with current rates and request volume. If the shorter prompt fails on an important case, restore the necessary instruction or example rather than optimizing for token count alone. This approach makes prompt optimization a measurable engineering practice instead of a one-time editing exercise.
For related workflows, review Prompt Evaluation Framework, Prompt Chaining Examples, and AI Model Comparison for Prompt Engineers.
