Optimization and precise evaluation of the inference process in Large Language Models (LLMs) is one of the main challenges for developers in the field of artificial intelligence. To achieve a proper understanding of model performance, eight vital aspects must be considered, including overall inference metrics, measurement of single requests, system synchronization, and memory analysis.
One of the crucial metrics in this area is latency, which represents the time elapsed to process a request from start to finish. Furthermore, analyzing GPU workloads using CUDA events, managing concurrent requests, and accurately calculating the cost per token are among the other main axes for evaluating model performance in single-machine environments and multi-machine clusters.