Inference is the process of running a trained model on new inputs to produce an answer or prediction, using the weights exactly as training left them.
How it works
A finished model loads its weights into memory and pushes each fresh prompt through them in a single forward pass, emitting one token at a time until the answer is complete. Batching groups many requests onto the same hardware while cached prefixes skip repeated work, so throughput climbs without any new learning. Training spends weeks shaping the weights while serving spends that investment in milliseconds per request, which is why cost per million tokens is the number every product team watches. Answers draw only on what the model already holds, so freshness and factuality rest on the training cutoff plus any retrieval layered on top.
Builders reach for inference tuning whenever answers must be fast and cheap: batching requests, caching shared prefixes, and serving compact models where full precision cannot pay its way.