The Demo vs. Production Gap
Prompt engineering tutorials show you prompts that work on the example. Production prompts need to work on the distribution — the full range of real user inputs, edge cases, adversarial inputs, and off-topic requests you'll encounter after you ship.
I've been building LLM-powered systems since GPT-3 — through talkr.ai's voice agents, manju's translation pipeline, and UAE Lead's conversational lead capture. What I've learned is that the techniques that matter most in production are almost never the ones that get the most attention online.
Structured Output Is Not Optional
The single highest-leverage change in any LLM integration: enforce structured output. If your system prompt asks for JSON and hopes the model delivers it, you will get malformed JSON in production. Guaranteed. It might be 0.1% of calls. It might be 5%. Either way, your pipeline breaks.
Use the model's native structured output mode (available in all major APIs now) or a library like Instructor/Outlines to constrain generation to a validated schema. The performance cost is negligible. The reliability gain is categorical — you go from "usually works" to "always returns a parseable object."
Design your schemas to be forgiving: use Optional fields for information the model might not have. Use enums for categorical outputs. Include a confidence field so downstream logic can handle uncertainty without guessing.
Chain-of-Thought: Use It Where It Matters, Skip It Where It Doesn't
Chain-of-thought prompting (asking the model to reason step by step before answering) genuinely improves accuracy on complex reasoning tasks. It also adds tokens — which means latency and cost.
My routing rule: CoT for decisions with more than 3 conditions, factual synthesis across multiple sources, and any output where an incorrect answer has high cost. Skip CoT for classification with clear categories, simple extraction, and latency-critical paths where accuracy is already above threshold.
In talkr.ai's agent system, we route to CoT dynamically: if the intent classifier confidence is below 0.85, we trigger a CoT reasoning pass before selecting the response strategy. Above 0.85, we respond directly. This cuts average token use by ~30% on the high-volume, simple-intent majority while preserving quality on the complex cases that need it.
Few-Shot Selection Over Static Examples
Static few-shot examples in your system prompt are better than no examples. Dynamic few-shot selection — choosing examples at runtime based on similarity to the current input — is substantially better.
The implementation: embed your example set. At inference time, embed the current input. Retrieve the K nearest examples. Inject them into the prompt. This is sometimes called "example-based prompting" or a lightweight form of RAG.
The gain is largest on tasks with diverse input types. manju's translation prompts use dynamic few-shot: a manga panel with sound effects gets examples of SFX translation; a UI screenshot gets examples of functional/concise translation. Same base prompt, different examples, dramatically better output distribution.
System Prompt Architecture
Long system prompts degrade. There's a well-documented phenomenon where instructions in the middle of a long context receive less weight than instructions at the beginning and end. If your system prompt is 2000 tokens of instructions, the instructions at token 800-1400 are underweighted.
My system prompt structure: persona + core constraints at the top (most important), task-specific instructions in the middle, output format specification at the bottom (second most important). Critical constraints appear both at the top and are repeated in brief at the bottom.
Keep system prompts under 800 tokens where possible. If you need more, restructure: move examples to dynamic few-shot, move reference material to RAG, move complex logic to tool calls. The system prompt should define who the model is and what it must never do. Everything else is retrieval.
Evaluation Before Optimization
The mistake I made most in early LLM work: optimizing prompts based on vibes. "This version feels better" is not a measurement. Build an evaluation set — 100-200 representative inputs with ground truth outputs — before you start optimizing. Run every prompt change against it. Track the number.
For talkr.ai, the eval set is 500 voice agent conversations with human-rated response quality. Every prompt change is measured against it before shipping. We've caught regressions that felt like improvements, and improvements that felt like regressions, in roughly equal measure. The eval set is the ground truth. Your intuition is a hypothesis.
Model Selection Is Also Prompt Design
Different models have different strengths that affect how you should prompt them. Claude excels at instruction following and structured output; its default verbosity needs to be explicitly constrained for concise outputs. GPT-4o is faster on structured tasks; it benefits from more explicit step decomposition. Gemini Flash is cost-effective for high-volume classification; its instruction following is strong but benefits from very explicit output constraints.
Don't port prompts between models without testing. A prompt optimized for one model will often produce degraded output on another. Treat model selection and prompt design as co-dependent decisions.