Most Generative AI models currently rely on GPUs or specialized accelerators, but recent findings suggest CPUs can also perform effectively for enterprise AI tasks. Google has tested Intel’s 4th-Gen Xeon processors, revealing that they can achieve reasonable latency when running large language models, like Llama 2, at 16-bit precision. The tests showed a time of 55 milliseconds per output token and a throughput of around 230 tokens per second in certain scenarios. While CPUs may not match the performance of high-end GPUs, they could be a cost-effective option, especially if organizations already have underused CPU resources. The choice between CPUs and GPUs ultimately depends on specific workload needs and budget considerations.
Title: Google’s Findings on CPU Performance in AI Workloads
In recent discussions about AI technology, many people have focused on GPUs as the go-to hardware for running AI models. However, Google has brought attention to the potential of CPUs, specifically Intel’s 4th-Gen Xeon processors, in handling AI tasks efficiently. Their recent experiments revealed that these CPUs can achieve acceptable performance levels for large language models (LLMs), which could change how businesses think about deploying AI solutions.
Google tested a model with 7 billion parameters using a virtual machine equipped with 176 virtual CPUs. They reported that the time it took to generate each output token was around 55 milliseconds. This performance level is quite promising, especially given that used the 16-bit precision settings during their tests.
Further, when it comes to processing multiple requests simultaneously, Google found that the CPU could manage about 230 tokens per second when running at a batch size of six. This means that for specific tasks, CPU-based computing can keep up with the demands of modern AI applications.
Interestingly, these tests explored the cost-effectiveness of CPUs compared to traditional GPUs. Although using CPUs can appear expensive at first—Google’s C3 instance costs around $5,464 a month—their findings suggested that switching to these advanced CPUs could offer significant speed improvements for fine-tuning AI models.
For organizations already investing in CPUs or with existing hardware, utilizing these advanced processors could provide an efficient alternative to GPUs, especially for smaller models. Google’s insights highlight a possibly overlooked area in AI computing, suggesting that CPUs should not be dismissed in favor of faster, but pricier, GPU options.
Looking ahead, even though CPUs may not match the performance of top-tier GPUs, they could still serve as a flexible, cost-effective solution for many AI workloads, especially in environments that already take advantage of existing computing power.
Tags: AI, CPUs, Intel Xeon, Google, AI performance, large language models, tech news
What does CPU-based AI mean?
CPU-based AI uses central processing units to run artificial intelligence tasks. It’s the traditional method for processing data, but it can be slower for complex AI models.
Why are the economics of CPU-based AI not great?
The main issue is that CPUs are usually slower than GPUs (graphics processing units) for AI tasks. This means CPU-based AI can take longer to process data and require more energy, making it less cost-effective.
What alternatives are there to CPU-based AI?
Other options include using GPUs and specialized hardware like TPUs (tensor processing units), which are designed to handle AI tasks more efficiently and quickly.
Is it still possible to do AI with just CPUs?
Yes, you can still run AI applications with CPUs, but they may not perform as well or as quickly as those using GPUs or other hardware designed for AI.
Should I avoid using CPU-based AI?
It depends on your needs. For smaller projects or less complex AI tasks, CPUs can still work well. For larger, more demanding AI applications, consider using GPUs or specialized hardware for better performance.
