Episode 15
In this episode of Tech Threads: Weaving the Intelligent Future, Baya Systems’ Nandan Nayampally sits down with Charlie Cheng, founder and CEO of TeRAM, to discuss the AI memory wall and the industry’s shift toward alternative memory architectures. With Nvidia licensing Groq’s technology and Cerebras drawing attention to wafer-scale computing, the race to bring memory closer to compute is reshaping how AI systems are built.
Charlie draws on a career spanning microprocessors, EDA, IP licensing, and ASICs, including his time as CEO of Kilopass, the memory IP licensor acquired by Synopsys. He explains why AI inference changes the memory problem: during decode, every token requires a pass through the entire model, so memory bandwidth now stands alongside capacity as a limiting factor in generating each profitable token. What was once an engineering concern has become a direct cost on the CFO’s P&L.
The conversation covers why on-die SRAM is no longer scaling, why HBM supply is constrained, and why 3D integration through hybrid bonding is the natural next step for the data center, on-prem AI, and edge devices such as AR/VR glasses. Charlie also shares TeRAM’s goal of delivering 64GB of memory capacity and about 200TB/s of bandwidth close to compute. That could shrink AI models from multiple racks to less than one and cut total power by roughly 7x.
As memory moves closer to compute, the pressure shifts to how data moves across the chip. Nandan and Charlie discuss how hybrid bonding and chiplets are changing network-on-chip topologies, partitioning, and routing, and why efficient data movement is the other half of breaking the memory wall.
Tune in for an expert look at how new memory architectures, chiplets, and efficient data movement could help unlock faster, more power-efficient, and more scalable AI inference.
