One day in AI is like one year in the real world.
Nowadays, whether it is the big model itself or the update speed of AI applications, it is simply hard to keep up.
Sora, Suno, Udio, Luma…blockbuster applications are released one after another.
As the survey data from InfoQ shows, although AIGC is still in its infancy, the market size has taken shape:
It is expected to reach RMB 450 billion by 2030. AIGC applications are blossoming in multiple areas, gradually penetrating from general scenarios to the depths of the industry.
The rapid development of the industry as a whole is certainly a good thing, but when it comes to the implementation of each application or large model, the competition is becoming increasingly fierce.
For example, a very fierce “price war” broke out among various large model manufacturers not long ago, with the main focus on who could be cheaper, and even directly pushing the price of large models into the “centimeter era”.
Coupled with the recent “supply cut” incident of OpenAI, domestic manufacturers have increased the intensity of giving away tokens while launching the “easy moving” plan.
The reason for this is closely related to the current trend of application being king, especially the trend of quickly implementing business at the lowest possible cost.
So the question is, how do big model players strike a balance between being fast, good and economical?
This brings us back to the unavoidable factor that accounts for the absolute majority of the cost – computing power.
When it comes to the training and reasoning of large models, many people's first reaction may be to think of GPU.
It is true that GPU has certain advantages in high performance, but its “flaws” are also quite obvious, namely insufficient supply and high price.
How to break the deadlock? Baidu Smart Cloud Qianfan Big Model Platform, the first-tier player in the domestic big model industry, has come up with a more cost-effective solution:
Except for a few large customers who pursue excellent performance of large models, most enterprises and institutions need to comprehensively evaluate the usage effect, performance and cost-effectiveness of large models when adopting them, which is the so-called “price ratio”.
Regarding the deployment of computing power, Xin Zhou, general manager of Baidu Smart Cloud AI and Big Model Platform, believes:
In fact, in the early days of AI, CPUs have always played a role; GPUs have only become popular in recent years.
In many scenarios, although GPUs have high-density computing capabilities, actual tests have shown that current high-end CPUs can also fully handle the task.
Moreover, the entire AI business flow not only requires large models to be calculated, but also involves early data cleaning and other links. The CPU plays a very important role in these links.
In a nutshell, in the era of big models, CPU is even more important than ever before, and it is one of the key factors that enable big models and applications to be implemented quickly, well and economically.
So what are the specific effects of “taking up the post”? Let's continue reading.
Domestic head large model player, picked CPU
There has been an explosion of AIGC applications in China, and Baidu Smart Cloud’s Qianfan large model platform has made an indelible contribution to this.
As a “one-stop” service platform for enterprises to use big models, Qianfan Big Model Platform has been used by more than 120,000 customers since its release in March last year, and has optimized 20,000 models and incubated 42,000 applications.
These applications cover many scenarios such as education, finance, office, and medical care, providing strong support for the industry's digital transformation.
In the field of education, Qianfan's large model platform enables applications such as test question generation, online grading, and question analysis, greatly improving the efficiency of teaching and exam preparation.
For example, users can provide reference materials, set the question type and difficulty, and the platform can automatically generate high-quality test questions. Interactive question analysis can provide personalized learning guidance based on each student's weak points.
In the office scenario, Qianfan Big Model Platform cooperates with leading companies in the industry to jointly create innovative applications such as intelligent writing assistants, which can quickly generate professional documents such as recruitment copy, marketing plans, data reports, etc. based on keywords entered by users.
It can also focus on various writing scenarios and can intelligently generate paper outlines, project reports, brand promotional materials, etc., greatly improving the work efficiency of administrative and marketing personnel.
Medical health is another major application track of Qianfan's large model platform. The model trained based on the medical knowledge base can automatically generate interpretations of physical examination reports, explain various indicators to users in easy-to-understand language, and provide personalized health guidance.
This allows the general public to better understand their own physical condition and achieve “self-health management”.
It can be seen that Qianfan's large model platform has realized the “last mile” implementation of AI models in multiple fields.
So how does Qianfan's large model platform support so many AI applications?
the answer is:Make CPU one of the options for customers and let the benefits of “price-performance ratio” benefit thousands of industries.
The reason for this is explained by Baidu Smart Cloud:
At present, there are still a large number of offline LLM application needs in the industry, such as generating article summaries, abstracts, data analysis, etc. Compared with online scenarios, offline scenarios usually utilize the platform's idle computing resources, and do not have high requirements for reasoning latency, but are more sensitive to the cost of reasoning. Therefore, users are more inclined to use low-cost and easily available CPUs for reasoning.
A large number of CPU-based cloud servers are deployed in cloud platforms such as Baidu Smart Cloud. Unleashing the AI computing power potential of these CPUs will help improve resource utilization and meet users' needs for rapid deployment of LLM models.
As for the effect, taking Llama-2-7B as an example, the output Token throughput on the fourth-generation Intel Xeon Scalable Processor can reach more than 100 TPS, which is a 60% increase compared to the third generation.

△Llama-2-7b model outputs token throughput
In low-latency scenarios, at the same concurrency, the first token latency of the fourth-generation Xeon Scalable Processor can be reduced by more than 50% compared to the third-generation Xeon.
After upgrading the processor to the fifth-generation Xeon Scalable Processor, the throughput can be increased by about 45% compared with the previous generation, and the first token latency can be reduced by about 50%.

△Llama-2-7b model first token delay
And the Qianfan large model platform team also said based on practical experience:
For LLM models with a scale below 30B, Intel Xeon Scalable Processors can be used to obtain a good performance experience.
In addition, by utilizing sufficient CPU resources, the demand for AI accelerator cards is reduced, thereby reducing the total cost of ownership (TCO) of LLM reasoning services, especially in offline LLM reasoning scenarios.
Moreover, on the Qianfan large model platform, there is not only its own ERNIE, but also many mainstream large models are integrated here.
This also confirms to a certain extent the performance of the fifth-generation Intel Xeon Scalable Processor.
Intel's fifth-generation Xeon, how to make performance and efficiency Pro Max?
Baidu Smart Cloud Qianfan big model platform not only requires big model inference as a workload, it is actually a platform covering the entire life cycle of big models.
Specifically, Qianfan Big Model Platform provides comprehensive functional services including data annotation, model training and evaluation, reasoning services and application integration, as well as rapid application orchestration and plug-in integration, to help big models be applied in multiple scenarios. In this way, it is a more cost-effective choice to make full use of the platform's widely deployed CPU resources than to deploy dedicated accelerators for big model reasoning.
For the large number of offline large model application needs on the Qianfan large model platform, such as generating article summaries, abstracts, and evaluating the effects of multiple models, these needs do not actually have high requirements for inference latency, but memory can easily become a bottleneck.
Using the CPU makes it easier to expand memory and can also utilize the computing resources of the platform when it is idle, further improving resource utilization and reducing the total cost of ownership.
In this context, the design of performance-intensive general computing application workloads (similar to P Core performance cores) in the fifth-generation Intel Xeon Scalable processors is particularly critical.
Compared with E Core (energy efficiency core), P Core adopts a design that pursues maximum performance. It can carry very heavy loads while also taking into account AI reasoning acceleration.
The fifth-generation Xeon Scalable Processor that adopts this design does not just talk about accelerating AI reasoning, but also optimizes the coordinated software and hardware, with all aspects taken into consideration.
In terms of hardware, Intel AMX (Advanced Matrix Extensions) technology is specially optimized for large-model reasoning and deep learning with a large number of matrix multiplication operations, which can be understood as the “Tensor Core in the CPU.”
With Intel AMX, the processor can complete up to 2048 INT8 operations per clock cycle, which is 8 times higher than the previous generation AVX512_VNNI instruction.
More importantly, the Intel AMX accelerator is built into the CPU core, bringing matrix storage and operations closer together. This feature can be applied to large model reasoning, reducing the latency of processing the next Token and providing a better experience for end users.

△Intel AMX can achieve AI acceleration more efficiently
In terms of software, Baidu Smart Cloud Qianfan Big Model Platform has introduced xFasterTransformer (xFT), a big model inference software solution deeply optimized for Intel Xeon Scalable Platform, and uses it as the backend inference engine. The main optimization strategies are as follows:
– Make full use of instruction sets such as AMX/AVX512 to efficiently implement core operators such as Flash Attention
-Use low-precision quantization to reduce data access and take advantage of INT8/BF16 operations
-Supports multi-machine and multi-card parallel reasoning for ultra-large-scale models

△Intel Xeon Scalable Processor LLM Reasoning Software Solution
Finally, I would like to add that choosing a hardware platform is not only related to the purchase price of the equipment itself, but also affects the subsequent maintenance costs and even the cost of talent reserves.
So as Baidu Smart Cloud said, cost-effective computing infrastructure, complemented by advanced large-model algorithms and platform software, allows upper-level developers to apply more smoothly and build their businesses, thus maximizing the commercial value of the cloud computing platform.
In the era of large models, CPU has great potential
Looking at the current situation, large models are moving from laboratories to industries, and from “toys” for a few people to “tools” available to the public.
This means that big model services must not only have excellent performance, but also be affordable and easy to deploy. In a nutshell, “fast, good, and economical” has become a key link in the commercialization of big models.
To achieve “fast, good and economical”, the choice of computing power infrastructure is crucial.
Traditionally, dedicated accelerators are the “standard configuration” for AI. However, with tight supply and high costs, the advantages of dedicated accelerators are waning.
In contrast, well-optimized high-end CPUs not only provide sufficient computing power to handle large model reasoning, but also have a broader deployment base, a more mature software ecosystem, and better security guarantees, and are beginning to be favored by more and more industry players.
The x86 architecture CPU represented by Intel Xeon series has a mature and complete software ecosystem and a broad application base. Millions of developers can use existing tools and frameworks to quickly build and optimize AI applications without having to learn additional specialized accelerator software stacks, greatly reducing development difficulty and migration costs.
At the same time, enterprise users can also use the multi-level security technology built into the CPU to achieve full-stack protection from hardware to software, fully ensuring data security and privacy. These advantages are difficult to match with current dedicated accelerators.
It can be seen that making full use of CPU for reasoning and turning AI from a “money-burning game” to “inclusive technology” is the key move for the AIGC industry to cope with the computing power threshold and promote the implementation of large-scale applications. In the future, with technological innovation and ecological improvement, this model will create value for more companies and inject new impetus into industrial development.
In addition to directly accelerating inference tasks, the CPU can also efficiently complete key steps in the entire AI process, such as data preprocessing and feature engineering, in a complete end-to-end AI pipeline. Various databases that support machine learning and graph analysis are also mainly built on the CPU. Taking the Xeon Scalable Processor as an example, in addition to Intel AMX, there are a series of data analysis engines such as Intel Data Protection and Compression Acceleration Technology (Intel QAT) and Intel In-Memory Analytics Accelerator (Intel? IAA) built into the processor, which makes better use of the CPU by offloading specific tasks, thereby improving the overall performance of the workload and accelerating data analysis.
It can be seen from this thatBuilding “fast, accurate, and stable” AI applications requires both the powerful computing power of dedicated accelerators and the CPU’s superb general computing capabilities to unleash the potential of the entire system.



