Part 1 — LLM Application Foundations

From software engineering to LLM engineering

From software to LLM engineering·Core·4 min read

Understand the one shift that reframes everything else in this book: you are no longer writing deterministic code whose correctness you can prove, but building a system around a probabilistic component whose quality you must measure. That shift is why evaluation is not optional.

If you come from software engineering, you already know how to build reliable systems: you write code, you reason about its behaviour, and you prove it correct with tests that pass or fail. Building with large language models keeps most of that discipline — and changes one thing at the centre that reshapes everything around it.

What you will understand by the end

  • Why an LLM is a probabilistic component, not a deterministic function.
  • Why you evaluate an LLM system on a distribution rather than proving it correct.
  • The new failure modes — confident-but-wrong, valid-but-meaningless — that ordinary tests don't catch.
  • Why reliability comes from the system around the model — a thread you pick up throughout the book.

The one shift

A function returns the same output for the same input, every time. You can unit-test it: one input, one expected output, green or red. An LLM does not work that way.

Key idea

An LLM is a probabilistic component. The same input can produce different outputs, and "correct" becomes "correct often enough, on the kinds of input you actually see." You stop proving behaviour and start measuring it — which is why evaluation is a first-class engineering activity, not an afterthought.

Two familiar habits have to be re-learned:

  • You program partly in prose. Alongside code, you steer the model with a prompt and the context you give it. The prompt is an interface you design, not a configuration you set once.
  • You can't assert one right answer. A test like assert f(x) == y doesn't fit a component that can phrase the same correct answer ten ways, or produce a wrong answer that looks plausible. Correctness is a rate over a dataset, established with an evaluation, not a single equality.

The new failure modes

Ordinary software fails loudly — an exception, a crash, a wrong-but-checkable value. LLM systems add failure modes that pass every type check and still be wrong:

  • Hallucination — a fluent, confident answer that is simply untrue.
  • Valid-but-wrong — output that satisfies the schema (parses fine, right shape) but means the wrong thing.
  • Silent drift — behaviour that changes when the model, prompt, or data changes, without any error being thrown.
Watch out

The most dangerous LLM bugs don't throw. A system can return perfectly formatted, confident output that is subtly incorrect — and no stack trace tells you. Catching those is the job of evaluation and observability, which is why half of this book is about them.

Mental model

LLM engineering = software engineering + a probabilistic component you must measure, not prove — wrapped in a system that makes it reliable.

That reliability comes from what you build around the model, so a measured result belongs to the whole configuration — the idea The model is only one component is built on.

Common mistakes coming from software engineering

  • Treating the model as a function to unit-test. One input/one output misses the distribution; you need a dataset and a scorer.
  • Trusting a demo. A prompt that works on three hand-picked examples tells you almost nothing about the other thousands.
  • Confusing "it ran" with "it's right." Valid output is not correct output.

Practical guidance

  • Design the prompt and context as an interface, and keep them under version control like code.
  • Decide how you will measure quality before you optimise it — a dataset and a scoring rule, not vibes from a demo.
  • Assume non-determinism: pin what you can (decoding settings), and treat run-to-run variation as something to measure, not wish away.
  • Build the system, not just the prompt — the reliability lives in the wrapper.

Summary

  • An LLM is a probabilistic component; correctness is a measured rate, not a proof.
  • New failure modes (hallucination, valid-but-wrong, drift) pass ordinary tests.
  • Reliability comes from the system around the model.
  • That is why this book teaches you to build these systems and to evaluate them — the two are inseparable.

Knowledge check

Why can't you unit-test an LLM the way you test a pure function?

A pure function is deterministic — one input, one output — so assert f(x) == y works. An LLM can produce different outputs for the same input and can phrase a correct answer many ways (or a wrong one plausibly), so a single equality can't capture correctness. You measure a rate over a representative dataset with a scoring rule instead.

What's the difference between "valid" and "correct" for an LLM's output?

Valid means it satisfies the expected structure (parses, right schema). Correct means it means the right thing. The two are independent axes — output can be perfectly valid and still semantically wrong, which is exactly the failure ordinary tests miss.

Related chapters