Cerebras Technologies has announced a significant update to its inference capabilities, achieving a remarkable 3x performance boost for its Llama 3.1-70B model. This advancement positions Cerebras Inference as a leader in speed and efficiency, enabling transformative applications across various industries.

Cerebras Technologies has recently unveiled the most substantial update to Cerebras Inference since its launch, achieving an impressive throughput of 2,100 tokens per second for the Llama 3.1-70B model. This performance enhancement represents a threefold increase over previous versions and establishes Cerebras Inference as a game-changing solution in the AI landscape.

The speed of Cerebras Inference is striking when compared to traditional GPU solutions:

• 16x faster than the fastest GPU solution available.
• 8x faster than GPUs running the smaller Llama 3.1-3B model.
• This performance leap is akin to a new generation of GPU upgrades (H100/A100) achieved through a single software release.

Fast inference is crucial for developing next-generation AI applications across various domains, including voice recognition, video generation, and advanced reasoning tasks. Leading companies like Tavus and GSK are already leveraging Cerebras Inference to enhance their workflows and push the boundaries of AI capabilities.

Cerebras Inference has undergone rigorous testing by Artificial Analysis, a third-party benchmarking organization. The results highlight its superiority:

• Cerebras Inference is 16x faster than the most optimized GPU solutions.
• It outperforms hyperscale cloud services by a factor of 68x.
• For multi-step workflows, it completes requests in just 0.4 seconds compared to 1.1 to 4.2 seconds on GPU-based solutions.

The latest release builds on previous advancements by optimizing critical kernels such as MatMul and element-wise operations. The Wafer Scale Engine has been enhanced to utilize peak bandwidth and compute capabilities more effectively. Additionally, speculative decoding has been implemented, allowing for faster answer generation while maintaining output accuracy.

The speed enhancements provided by Cerebras Inference are already transforming how organizations approach AI application development:

• Pharmaceutical Research: GSK is utilizing the speed of Cerebras Inference to develop intelligent research agents that significantly enhance productivity in drug discovery processes.
• Voice AI Development: LiveKit's CEO notes that with Cerebras Inference, voice AI can now operate at human-level speed and accuracy, revolutionizing real-time applications.
• Reasoning Tasks: The platform enables models to perform extensive reasoning without incurring typical latency penalties, making it ideal for complex coding and research tasks.

The substantial performance improvements showcased in this update demonstrate the potential of the Wafer Scale Engine for inference tasks. As Cerebras continues to optimize both software and hardware capabilities, users can expect further enhancements in model selection, context lengths, and API features in the near future.

The recent advancements in Cerebras Inference highlight the transformative power of fast inference in AI applications. With a throughput of 2,100 tokens per second for Llama 3.1-70B, Cerebras has set a new standard for performance that will enable developers to create more responsive and intelligent applications across various sectors.

You can read official Cerebras blog post where there are more details about the results included.