SSCS December Technical Webinar: How Close Should Compute and Memory Be?

Abstract: Compute-in-memory (CIM) and processing-near-memory (PNM) have emerged as promising approaches to overcome the limitations of architectures that separate processing from memory. While this bottleneck is often attributed to von Neumann’s design, such critiques overlook its historical significance: the separation of logic and memory enabled the decoupling of software from hardware, laying the foundation for general-purpose computing. In this talk, we explore three key considerations in response to the question: How close should compute and memory be? First, memory and logic technologies follow distinct optimization paths. While integrating logic with memory may reduce local data movement, it may also compromise memory density, leading to more off-chip access and lower overall efficiency. Maintaining a degree of separation enables each technology to optimize its strengths, which can lead to greater overall efficiency. Second, tightly integrating compute and memory may constrain logic complexity due to limited area within the memory core, reducing architectural adaptability. As machine learning models continue to evolve, rigid architectures may reduce flexibility, thus hindering their adoption. Architectural flexibility is therefore essential to support diverse and rapidly advancing AI models. Third, current systems and applications are deeply rooted in established architectures, with substantial investment in software infrastructure and development platforms. Meaningful progress toward new paradigms requires coordinated efforts across hardware, software, and domain expertise. We hope to raise awareness of these three areas as researchers continue to innovate toward addressing the energy challenges posed by artificial intelligence.