Unleashing Heterogeneous Compute: Lessons from Real-World System Design

Brian Carlson

Introduction

Joint solution reduces risk and accelerates time-to-silicon for advanced semiconductor platforms

This blog summarizes the key points from their presentation.

The Challenge of Heterogeneous Compute

The presentation opened with a fundamental principle: “If you can’t feed it, you can’t use it.” In modern System-on-Chip (SoC) designs, the abundance of compute resources—more cores, GPUs, and acceleration blocks—does not automatically translate to better performance. The bottleneck lies in efficiently moving data, synchronizing execution, and managing memory. The speakers highlighted that true heterogeneity is not just about integrating multiple compute engines but ensuring they cooperate architecturally. Key factors influencing performance include:

These elements collectively determine how well a system can handle data-intensive workloads, a critical consideration for heterogeneous platforms.

Cache Analysis as a Diagnostic Tool

The core of the presentation revolved around a series of controlled profiling experiments using CacheStudio, a tool developed by Baya Systems. These experiments analyzed cache behavior across L1, L2, and L3 cache levels in CPU-only, GPU-only, and mixed CPU+GPU workloads. The goal was to uncover behavioral patterns—such as stalls, scalability, and saturation points—rather than merely benchmarking absolute performance. By varying cache sizes (e.g., L1 at 16 KB and 32 KB, L2 at 64 KB, 128 KB, and 256 KB), the team isolated the impact of cache configurations on hit rates and memory load.

The analysis revealed how different cache sizes and configurations affect system performance, particularly in terms of contention, reuse, and locality mismatch. For instance, the GPU cache analysis on L2 showed hit rates varying significantly with L1 and L2 size combinations, with specific data points indicating a hit rate of 45% at 16 KB L1 and 64 KB L2. These findings underscored the importance of tailoring cache configurations to specific workload demands to optimize throughput and minimize bottlenecks.

Key Insights and Design Learnings

The presentation emphasized that cache analysis serves as a proxy for understanding broader system-level dynamics. It makes “invisible” issues—such as contention, poor data locality, and interconnect inefficiencies—visible to system architects. The key takeaway was the need for early system-level profiling to shape architecture design, rather than focusing solely on cache sizing or adding more compute engines. The speakers stressed that performance in heterogeneous systems stems from how compute engines interact, not just their raw computational power.

Specific design learnings included:

These insights are broadly applicable to any data-intensive heterogeneous platform, from edge devices to high-performance computing systems.

Closing Thoughts

The presentation concluded with a reiteration of the guiding principle: “If you can’t feed it, you can’t use it.” Cache modeling was presented not as the end goal but as a tool to inform architectural decisions. By profiling systems early, designers can address fundamental issues in data flow and resource coordination, leading to more efficient and scalable heterogeneous architectures. The session underscored the importance of a holistic approach to system design, where cooperation between compute engines is prioritized over merely increasing their number.

Conclusion

About Baya Systems