Working with LLMs taught me how to run a kitchen

Shipping one LLM feature is easy.

Shipping the tenth is where most teams hit a wall.

That was our story at Ailu. We had good use cases, real demand, and fast delivery in the beginning. But as features grew, our process started to break.

The best way I found to explain it is a kitchen.

If every order is made from memory, by one person, with no station setup, things work only while volume is low. Once demand increases, quality drops, timing drifts, and nobody can tell where the mistake happened.

LLM systems behave the same way.

A production LLM feature is a process

A lot of teams still treat LLM work like this: send one prompt, get one answer, move on.

That can work for a demo. It does not hold in production.

In production, each request usually needs multiple actions:

• Understand the intent
• Pull the right context
• Apply business rules
• Generate an output
• Validate and log what happened

That is not a single prompt. That is a process.

Why manual prompting broke for us

In the first releases, manual prompting felt fast. We shipped quickly and learned a lot.

Then complexity compounded.

Each new feature came with slight variations of prompt style, retrieval logic, and output handling. Soon we had multiple patterns solving the same problem in different ways. When quality dropped, nobody had a full map of what had run.

The typical symptoms were clear:

• Logic got scattered across teams and contexts
• Feature velocity slowed down after each new launch
• Debugging became mostly guesswork
• Prompts became hardcoded and risky to evolve

The model matters, but the workflow determines whether the result can be repeated, inspected, and repaired.

The architecture we chose: steps and pipelines

We rebuilt the system around two concepts:

• Step: one isolated action with clear input, output, and responsibility
• Pipeline: an ordered sequence of steps that resolves one business flow

This boundary sounds simple, but it changed everything.

A step could be replaced without rewriting the full flow. A pipeline could grow without turning into spaghetti. Teams could reason about a single part of the system without opening ten files and three dashboards.

On top of this, we built a visual Workflow Builder so non-dev roles could assemble and adjust flows with governance instead of dependency on release cycles.

How execution worked in production

We executed pipelines in two modes:

• Workers for queued and asynchronous processing
• Product triggers for event-driven and near-real-time flows

This split gave us control over throughput and user experience.

Some jobs could run in background with retry policies. Others needed immediate reaction after a user action. We treated execution mode as a product decision, not an implementation detail.

A few operational rules made this stable:

• Idempotency on critical steps, so retries did not duplicate side effects
• Timeouts and bounded retries per step
• Dead-letter handling for failed runs
• Correlation IDs from start to finish for traceability

Without this layer, failures look random. With this layer, failures become diagnosable.

Prompt engineering stopped being "copywriting"

As soon as we moved from prototypes to product flows, prompt engineering became a systems concern.

We stopped treating prompts as ad hoc text and treated them as versioned assets. Each prompt had a role in the pipeline, expected inputs, output contract, and change history.

Some practices that helped us a lot:

• Separate instruction, context, and formatting expectations
• Keep prompts explicit about constraints and allowed sources
• Standardize output shapes for downstream validation
• Version prompts and roll changes gradually

This reduced regressions when multiple people edited prompts and made it possible to compare versions with real metrics instead of opinions.

Context engineering: less stuffing, more curation

Another lesson: most failures were context failures, not model failures.

When context is noisy, even a strong model underperforms. When context is curated, smaller models can produce excellent output.

We moved from "dump everything in the prompt" to context assembly by policy:

• Retrieve only relevant chunks
• Re-rank when needed
• Remove duplicates and stale artifacts
• Cap context by token budget and business priority

Quality improved while latency and cost came down.

Model routing became an architecture decision

One practical lesson: model selection should happen per task, not per company preference.

We used OpenAI for most of our production volume. We used Anthropic models in specific tasks where their behavior matched the requirement better. For image-related steps, we used Gemini.

The key is to benchmark per step, not by hype.

In several cases, a cheaper model performed better for a specific stage than a more expensive one. Better fit, lower latency, lower cost.

We started using a policy matrix per step:

• Required quality threshold
• Latency budget
• Cost ceiling
• Failure fallback strategy

That made model choice auditable and easier to evolve over time.

Tokenizer count changed our cost curve

Another decision that paid off was tracking token count before execution.

We added token budgeting at step level:

• Estimate token usage before sending context
• Trim, summarize, or restructure payloads near the limit
• Route to a different model when cost-benefit made more sense
• Record projected vs real token usage for calibration

This gave us tighter spend control and fewer failures due to oversized context. More importantly, it improved predictability, which is essential when usage scales fast.

Evaluation and quality gates

One thing that surprised us: "looks good" is a weak quality metric.

We needed repeatable evaluation.

So we defined test sets by flow and tracked outcomes over prompt/model versions. For high-impact paths, we added quality gates before full rollout.

Our evaluation stack combined:

• Offline eval sets with representative cases
• Rule-based checks for format and safety constraints
• Human review on ambiguous classes
• Gradual rollout with rollback criteria

This changed discussions from taste to evidence.

Observability was non-negotiable

LLM failures are often silent. You get a valid response that is semantically wrong.

If you do not trace the run, you cannot explain the failure later.

We logged each pipeline execution end-to-end:

• Step-by-step timing
• Input and output metadata
• Model used at each call
• Token usage and estimated cost
• Retrieval artifacts and selected context
• Final status and error classification

That gave us a reliable path for debugging, cost governance, and continuous improvement.

Guardrails and failure handling

Reliability improved when we stopped assuming "one pass" success.

We added structured guardrails:

• Output validation before side effects
• Safe fallbacks when a step failed
• Explicit escalation paths for low-confidence outputs
• Retry only where retry made semantic sense

This reduced catastrophic errors and made the system resilient under bad inputs and partial outages.

What changed after the rebuild

Once the workflow system was in place, we felt the difference quickly:

• Less manual work
• More orchestration
• Better consistency across features
• Faster debugging and incident response
• Better cost control without quality collapse
• More autonomy for non-engineering stakeholders

In short: more precision at scale.

When this approach is worth it

If your team is shipping one isolated AI feature, keep it simple.

If your product depends on multiple AI capabilities, dynamic context, and ongoing iteration, this architecture pays back quickly.

A practical rule we used:

• One feature: optimize for speed
• Multiple features: optimize for system design

What a step contract really includes

When people hear "step," they usually imagine a simple box in a pipeline diagram. In practice, each step needs a contract that can survive production traffic, team changes, and new requirements.

For us, every step eventually had five explicit dimensions:

• Functional objective: what this step must do, and what it must never do
• Input contract: expected fields, required context, and hard constraints
• Output contract: expected shape, optional fields, and failure surface
• Runtime policy: timeout, retry behavior, idempotency, and fallback rules
• Observability fields: what to log so the run remains debuggable later

This sounds heavy on paper, but it reduced integration friction a lot. Once these contracts existed, teams could swap implementations with much lower risk. A retrieval step could be improved without forcing downstream changes. A generation step could be routed to a different model without breaking consumers. A validation step could become stricter without creating silent regressions.

The hidden value of strong contracts is team velocity. People move faster when boundaries are clear and failures are explicit.

The pipeline pattern we reused the most

Across different product features, we noticed most pipelines converged to a similar structure. Not identical logic, but a repeatable skeleton:

• Intake and normalization
• Context assembly
• Decision or generation
• Validation and policy checks
• Post-processing and side effects
• Trace persistence and metrics

This "shape" became our default mental model. It gave everyone a shared language for design reviews and incident response. If a run failed, we quickly knew where to look. If quality dropped, we could ask whether the problem was retrieval, instruction quality, or post-processing constraints.

The biggest gain was reducing random architecture decisions. New flows started from a proven structure, then specialized only where needed. That prevented overfitting each new feature to whoever happened to implement it first.

In kitchen terms, every station had its role. Mise en place came first, cooking came after, plating came at the end. You can innovate in recipes, but you do not remove the fundamentals that keep service stable.

Prompt engineering became lifecycle management

At prototype stage, prompt engineering is usually treated like trial and error. In production, that approach breaks quickly because prompts are part of business logic.

We moved to a lifecycle mindset:

• Draft a prompt with clear intent and boundaries
• Validate against a representative sample
• Run a controlled rollout
• Measure quality and cost impact
• Keep version history with rationale

A key lesson was separating three layers inside prompt design:

• Policy layer: non-negotiable constraints, role, and behavioral boundaries
• Task layer: objective for that specific step
• Formatting layer: output shape and parsing expectations

When these layers were mixed in one long paragraph, updates caused regressions. When they were explicit, updates became safer and easier to review.

We also stopped evaluating prompts only with "does it read well?" and started asking harder questions. Does it fail safely? Does it degrade under missing context? Does it produce stable output over repeated runs? Does it comply with downstream contracts? Those are the questions that matter when your system is part of core operations.

Context engineering was where most quality gains came from

Most teams overestimate model selection and underestimate context quality. We made that mistake too.

Early on, when results were weak, we assumed we needed a stronger model. In many cases, the real issue was context assembly. Irrelevant documents, duplicated snippets, stale records, or conflicting policy text were polluting the request.

We shifted from "maximum context" to "minimum sufficient context." That required better retrieval discipline:

• Better chunking based on meaning instead of fixed size alone
• Metadata filters aligned with business domain
• Re-rank logic to prioritize high-signal evidence
• Deduplication and recency control
• Hard caps by token budget and step priority

This consistently improved answer quality and reduced spend. It also improved trust internally, because teams could inspect why a response was produced and what evidence was used.

In practical terms, context engineering became the equivalent of ingredient sourcing. Better ingredients reduce the burden on the chef. Better context reduces the burden on the model.

Routing policies and fallback trees

Model routing started as a cost optimization and ended up as a reliability strategy.

At first, we routed by rough intuition. Later, we formalized routing policies per step with explicit thresholds:

• Expected quality floor for that step
• Target latency budget
• Cost ceiling per run class
• Allowed fallback sequence

This gave us deterministic behavior under stress. If one model degraded, timed out, or became too expensive for a class of requests, the system could switch according to policy instead of ad hoc edits.

We also learned that fallback trees should not be unlimited. Too many retries and fallbacks increase latency, cost, and confusion. Bounded fallback paths with clear stop conditions worked better in production.

The combination of primary model plus controlled fallback created more stable service than any single-model strategy we tested. It also made vendor diversity practical. We used OpenAI for most volume, Anthropic where specific behavior was stronger for the task, and Gemini in image-related steps, all behind policy-driven routing instead of hardcoded assumptions.

Latency engineering for multi-step systems

Once flows became step-based, total latency became a composition problem.

A pipeline with eight good steps can still feel slow if each step is slightly inefficient. We started managing latency as a budget distributed across the pipeline, not as a single endpoint metric.

A few practices helped:

• Define target latency per flow class, then assign budget per step
• Run independent steps in parallel when business rules allowed
• Cache deterministic intermediate outputs when safe
• Use streaming where user experience benefited from progressive response
• Fail fast on invalid inputs before expensive operations

We also separated user-facing latency from backend completion latency. In some flows, returning an acknowledgment quickly and finishing enrichment asynchronously gave better product experience without sacrificing quality.

The main insight: latency is an architecture property. You cannot fix a slow pipeline only by changing one model parameter at the end.

Cost governance as an operating ritual

Token spend can grow quietly until it becomes a business problem. We avoided this by making cost review routine instead of reactive.

Every week, we reviewed cost at three levels:

• Per flow: which pipelines were drifting
• Per step: where token spikes happened
• Per release: what changed after recent prompt or routing updates

This cadence made optimization continuous. Small adjustments in context size, prompt structure, and routing often had bigger financial impact than large one-off interventions.

Tokenizer count was central here. Estimating before execution and comparing with actual usage gave us tighter forecasts and fewer budget surprises. Over time, we built better intuition for which features were naturally expensive and which were expensive because of design mistakes.

Cost control stopped being "cut quality" and became "remove waste."

Evaluation without fooling ourselves

Evaluation is where many teams create false confidence. We did too, initially.

If you evaluate only on easy examples, almost every version looks good. If reviewers know which version they are seeing, bias appears. If you run tiny samples, variance dominates and conclusions are noisy.

We improved this with a practical evaluation discipline:

• Maintain representative datasets per major flow
• Include hard and ambiguous cases alongside ideal inputs
• Compare versions on the same sample and conditions
• Track failure categories alongside pass rate
• Re-run evaluation after major retrieval or policy changes

For sensitive flows, we combined automated checks with targeted human review. Automation gave scale and repeatability. Human review captured nuance where strict rules were insufficient.

The most useful change was categorizing failures by root cause: retrieval gap, instruction ambiguity, policy conflict, formatting break, or model behavior drift. That made improvement work specific and actionable.

Incident response and what changed in practice

Two incidents shaped our design more than any architecture meeting.

In the first, output quality dropped for a high-traffic flow right after a prompt update. The response format still passed basic checks, so the issue stayed undetected for hours. We fixed this by adding stronger semantic checks and phased rollout with explicit kill switches.

In the second, a retrieval policy change increased context size and pushed some runs near model limits. Latency and failure rate climbed gradually instead of failing hard. We fixed this with stricter token pre-checks, dynamic context truncation, and tighter latency alerts per step.

Both incidents reinforced the same point: production systems fail at boundaries, not at the center. Guardrails need to cover those boundaries proactively.

Team design mattered as much as system design

The workflow engine changed both the system and the way the team worked.

Before this structure, ownership was blurry. Product asked for output improvements, engineers changed prompts, data teams tuned retrieval, and nobody had end-to-end accountability.

With step-based pipelines, ownership became clearer:

• Product owned target behavior and acceptance criteria
• Engineering owned pipeline reliability and contracts
• AI specialists owned prompt and model strategy
• Operations owned monitoring thresholds and incident playbooks

This reduced cross-team friction because responsibilities were explicit. It also improved onboarding. New people learned the system faster when they could map responsibilities to concrete stages in a pipeline.

Technical architecture and team architecture reinforced each other.

Migration path from ad hoc calls to a workflow system

One concern we hear often is whether this requires a full rewrite. In our experience, no.

A safer migration path is incremental:

1. Identify the highest-impact flow with recurring quality or cost pain
2. Extract that flow into step boundaries without changing behavior first
3. Add tracing and token budgeting
4. Introduce routing policy and prompt versioning
5. Roll out gradually and measure
6. Repeat for the next flow

This approach keeps momentum while reducing risk. Teams get early wins and build internal confidence before expanding the architecture.

The mistake is trying to redesign everything at once. Workflow systems should be adopted as operational upgrades, not as a big-bang rewrite.

Common anti-patterns we avoided later

Looking back, several anti-patterns created avoidable pain:

• Treating prompts as static strings hidden in product code
• Using one model for every task regardless of requirements
• Maximizing context size instead of relevance
• Adding retries without clear stop conditions
• Logging too little to reconstruct failures
• Optimizing only for demo quality, not production stability

None of these fail immediately. That is why they are dangerous. They seem fine in the first weeks, then compound quietly as feature count grows.

The healthier pattern is to optimize for maintainability early: explicit contracts, controlled execution, measurable quality, and policy-based decisions.

The operating checklist we kept close

For teams building similar systems, this checklist helped us stay disciplined:

• Every step has explicit input and output contracts
• Every flow has a latency budget and cost budget
• Prompts are versioned and evaluated before broad rollout
• Token count is estimated before execution
• Model routing is policy-driven, not ad hoc
• Traces are complete enough for incident reconstruction
• Fallback behavior is bounded and intentional

It is not glamorous work, but this is what made our AI stack dependable.

What held up in production

If AI is central to the product, define the stations, handoffs, service standards, and recipes before peak traffic. That operating discipline is what made LLM delivery sustainable for us.