Inference

Why This Matters

Training happens once in a lab; inference happens billions of times a day in your pocket. Every chatbot reply, face unlock, and fraud check is an inference. For any AI product, inference is where the costs accumulate, the latency is felt, and the user experience is won or lost. When engineers talk about "serving," "tokens per second," or "GPU bills," they are talking about this lesson.

Everyday Analogy

Medical school versus seeing patients. Training a doctor takes a decade and a fortune, and it's done once. Then the trained doctor diagnoses patient after patient, each visit taking minutes and costing a consultation fee. Inference is the consultation: fast, repeatable, priced per visit, and only possible because of the long training that came before. No learning happens during the visit; the doctor applies what they already know.

Training vs Inference: The Two Lives of a Model

One asymmetry defines the industry: because inference just reads frozen weights, it parallelizes beautifully, runs on modest hardware, and can be priced per use. The entire AI API economy (AI-059's token pricing) is inference economics.

What Happens During One Inference

Your phone camera frames a cat; the photo app labels it. In ~20 milliseconds:

1. Preprocess. The photo is resized and converted to a grid of numbers.
2. Forward pass. The numbers flow through the network's layers — multiply by weights, add, squash, layer after layer (the arithmetic from AI-004), with no weight changes.
3. Output. The final layer emits probabilities: cat 0.94, dog 0.04, other 0.02.
4. Postprocess. The app takes the top label and tags your photo.

For an LLM, the same forward pass produces probabilities over the next token, then runs again for the next token, and again, which is why long responses stream word by word and why output tokens cost more than input (AI-059).

The Three Numbers That Rule Serving

Production inference is managed by three metrics:

• Latency: how long one request takes. Users feel anything over ~200ms in interactive apps; LLM chat lives at 1–10s. Techniques: smaller models, quantization (AI-065), caching, streaming.
• Throughput: requests served per second per machine. Batching many requests through the GPU together multiplies it.
• Cost per request: GPU-seconds burned per prediction. At a million requests a day, shaving 30% off inference cost is real money (AI-039 and AI-054 go deep).

These three trade against each other: bigger batches raise throughput but add latency; smaller models cut cost but may cut quality. Serving engineering is choosing your point on that triangle.

Where Inference Runs

The push toward on-device (Apple Intelligence, phone-scale models) is powered by the compression techniques of AI-065: distill and quantize until the model fits in your pocket.

Real-World Showcase

• OpenAI, Anthropic, and Google each serve billions of LLM inferences daily. Inference fleets, not training runs, are their biggest ongoing compute cost.
• Visa runs fraud inference on every transaction with a latency budget of milliseconds; the model must answer before the payment terminal times out.
• vLLM, an open-source serving engine, can triple LLM throughput purely by scheduling and batching smarter, on the same model and same GPU, just better inference engineering (AI-066).

Try It Yourself

1. Feel the latency spectrum: unlock your phone with your face (~50ms, on-device), then ask a chatbot a question and watch tokens stream (1–10s, cloud LLM). Both are inference; hardware, model size, and distance explain the difference.
2. Estimate a serving bill: a support bot uses 4,400 input + 400 output tokens per conversation (the AI-059 worked example). At $3/M input and $15/M output, verify one conversation ≈ $0.019, then scale to your imagined traffic.
3. Find the batch effect: ask a chatbot the same question at a quiet time vs a busy time and notice latency variance. You are feeling the throughput/latency trade-off of shared serving infrastructure.

Common Mistakes

• Confusing inference with learning: the model is frozen, and your chat does not train it (context ≠ learning, see AI-061).
• Ignoring inference cost while obsessing over training cost. At product scale, serving dominates the bill.
• Benchmarking quality but not latency: a brilliant model that answers in 30 seconds loses to a good one that answers in 2.
• Assuming inference needs the same hardware as training, when quantized models serve on far cheaper chips (AI-065).
• Forgetting the pipeline. Production inference includes pre/postprocessing, safety filters, and retries, not just the model call (AI-021, AI-030).

Key Takeaways

• Inference = read-only forward pass through frozen weights; training writes, inference reads.
• It is the phase users experience and businesses pay for, request by request.
• Latency, throughput, and cost per request are the three ruling metrics, and they trade off.
• Inference runs everywhere from cloud GPUs to your phone; compression moves it down the stack.
• LLM streaming, token pricing, and GPU bills are all inference phenomena.