A couple of years ago, "AI engineer" mostly meant someone who could wire an API call into a chat box. Today the title covers people shipping search, support automation, coding tools and fully autonomous agents into production. The models got dramatically better — but the hard part moved. It's no longer can the model do it? It's can you build a system around the model that does it reliably, cheaply and safely?
This post is my working definition of the job, the stack I think about, and the habits that separate a demo from a product.
The one-sentence definition
AI engineering is product engineering where one of your components is probabilistic.
Everything else follows from that. A database returns the same row every time; a model doesn't. So you need the same things you'd build around any unreliable dependency — contracts, tests, observability, fallbacks — plus a few that are unique to language models: context design, evals and cost control.
The stack, layer by layer
- Model — which model, at what effort level, for which route. Not every request needs your most capable model.
- Context — the system prompt, retrieved documents, conversation history and tool results. This is where most quality is won or lost.
- Tools — the actions the model can take: search, database queries, code execution, your own APIs.
- Output contracts — structured outputs and validated schemas, so downstream code never parses free text.
- Evals — a test suite for behaviour, run on every prompt or model change.
- Guardrails & observability — logging, tracing, refusals, rate limits, budgets and human review where the cost of an error is high.
Notice that "prompt" is only one part of one layer. Treat it as source code: version it, review it, test it.
Context engineering beats prompt engineering
Modern models follow instructions well. The failure mode I see most often isn't a badly worded prompt — it's the model being given the wrong information, too much irrelevant information, or none of the information it needed. A few rules I follow:
- Put stable content first (system prompt, tool definitions, reference docs) and volatile content last. That's also exactly what makes prompt caching work.
- Retrieve less, but better. Ten precise passages beat a hundred vaguely related ones.
- Say why, not just what. "Answer in under 80 words because this renders in a mobile notification" generalises better than "be brief".
- Give the model a way out: "If the documents don't contain the answer, say so." Most hallucinations are the model trying to be helpful with nothing to go on.
Evals before vibes
The single biggest upgrade you can make to an AI feature is a small, honest eval set. Twenty to fifty real inputs with a clear pass/fail check is enough to start. Without it, every prompt tweak is a guess and every model upgrade is a gamble.
Here's the shape of the smallest useful eval harness, using the Anthropic TypeScript SDK:
import Anthropic from "@anthropic-ai/sdk";
import cases from "./cases.json" with { type: "json" };
const client = new Anthropic(); // reads ANTHROPIC_API_KEY
type Case = { input: string; mustInclude: string[] };
async function run(c: Case) {
const res = await client.messages.create({
model: "claude-opus-5-5",
max_tokens: 16000,
system: "You are a support assistant for Acme. Answer from the docs only.",
messages: [{ role: "user", content: c.input }],
});
const text = res.content
.flatMap((b) => (b.type === "text" ? [b.text] : []))
.join("\n")
.toLowerCase();
return c.mustInclude.every((s) => text.includes(s.toLowerCase()));
}
const results = await Promise.all((cases as Case[]).map(run));
const passed = results.filter(Boolean).length;
console.log(`${passed}/${results.length} passed`);
process.exit(passed === results.length ? 0 : 1);String matching is crude, and that's fine on day one. As the suite grows you'll add stricter checks — schema validation, exact-match on extracted fields, or a second model grading against a rubric. What matters is that it runs in CI and fails loudly.
Cost and latency are design constraints
A feature that's brilliant but costs a dollar a request, or takes forty seconds, isn't shippable. The levers, in the order I reach for them:
- Prompt caching — cache your long, stable prefix. It's close to free money.
- Trim the context — every irrelevant token costs money and attention.
- Tune effort — the newest models let you trade thinking depth for speed per request. Simple routes rarely need the top setting.
- Pick the right model per route — but measure first. A more capable model at lower effort often beats a smaller model at higher effort.
- Batch what isn't interactive — overnight summarisation doesn't need real-time pricing.
Measure cost per completed task, not cost per request. A cheaper call that needs three retries isn't cheaper.
Ship like it's a distributed system
Because it is. Your model provider is a remote dependency with rate limits, timeouts and occasional refusals. Handle them explicitly: typed errors, retries with backoff for 429s and 5xxs, streaming for long outputs, and a clear fallback when the model declines or the call fails. Log inputs, outputs, token usage and latency for every call — you can't debug what you didn't record.
How to get good at this
- Build one small thing end to end and put it in front of real users.
- Write the eval before you write the second prompt.
- Read the provider docs properly — the APIs change fast, and last year's patterns are often deprecated.
- Study failures, not successes. Your logs are the curriculum.
AI engineering rewards the same instincts as any good engineering: small reliable pieces, tight feedback loops and honest measurement. The model is the exciting part — the system around it is the job.