Latency
How long the model takes to reply — typically 1-30 seconds depending on length. High latency can disrupt real-time client interactions if the FA is waiting for a response during a meeting.
Latency refers to the response time of an AI model, typically ranging from 1 to 30 seconds based on output length. In professional settings, high latency is a critical performance metric because it can cause significant friction during real-time client interactions.
Latency is the temporal delay between the submission of a prompt and the completion of the generated output, measured in seconds as a function of computational load and token sequence length. In operational contexts, this duration—typically 1-30 seconds—must be minimized to prevent latency-induced disruption during synchronous client-facing interactions.
evolution
- 1956 · historyDartmouth Workshop
The concept of computational response time was first implicitly addressed as a bottleneck for real-time AI problem solving.
- 2012 · historyAlexNet Breakthrough
The rise of deep learning on GPUs shifted the focus of latency from algorithmic complexity to hardware-accelerated inference speed.
- 2022 · historyLLM Mainstream Adoption
The release of ChatGPT highlighted 'Time to First Token' (TTFT) as a critical metric for user experience in conversational AI.
- 2023 · historySpeculative Decoding
New inference optimization techniques emerged to reduce latency by using smaller models to predict tokens for larger models.