Inference

Executive Definition: Inference — The process of running live data through a trained machine learning model to make a prediction or generate an output.

After a model is trained, it enters the inference phase where it is deployed to process new, unseen data. In production environments, inference must be optimized for speed, throughput, and cost-efficiency, often requiring specialized hardware or model compression techniques.

Frequently Asked Questions

How is inference different from training?

Training is the computationally intensive process of teaching a model using a large dataset. Inference is the application of that trained model to new data to generate quick predictions or responses.

What drives inference costs?

Inference costs are determined by model size, input/output sequence length, hardware requirements (like GPUs), and the volume of requests. Techniques like quantization can help reduce these costs.

What is inference latency?

The time it takes for a model to process an input and generate a response. Low latency is critical for real-time applications like chatbots or autonomous driving.