What is AI Inference?

AI inference is the process of running a trained model to generate predictions or outputs from new input data. Unlike training (which is done once), inference happens every time a user interacts with an AI feature — every chatbot response, every code suggestion, every image generation.

Inference cost is the dominant variable cost in AI features. Training GPT-4 cost an estimated $100M, but inference costs across all users dwarf that number. Each inference call consumes GPU compute proportional to model size and input/output length.

Inference optimization is a critical engineering discipline: model quantization (reducing precision from 32-bit to 8-bit or 4-bit), batching (processing multiple requests simultaneously), caching (storing common responses), and distillation (creating smaller student models from larger teacher models).

For product leaders, inference cost is the unit cost that determines whether your AI feature has positive or negative unit economics. Richard Ewing's AUEB tool calculates Cost of Predictivity — the true per-query cost including inference, retrieval, verification, and error handling.