In this episode of the Computer Architecture Podcast, hosts Dr. Suvinay Subramanian and Dr. Lisa Hsu engage in a captivating discussion with Dr. Bill Dally, Chief Scientist and Senior Vice President of Research at Nvidia, as well as a Professor of Computer Science at Stanford University. Dr. Dally, renowned for his extensive contributions to parallel computer architectures, interconnection networks, GPUs, and accelerators, shares his insights on the evolving landscape of computing.
The conversation delves into the future of computing innovation in the post-Moore era, with a particular focus on domain-specific accelerators. Dr. Dally elaborates on the necessity of these specialized hardware components as traditional serial processors reach their performance limits. He discusses the design philosophy behind effective domain-specific accelerators, emphasizing the critical role of reprogramming applications to be hardware-friendly and highly parallel. The episode also touches upon the challenges and strategies for successful technology transfer from research to product, the dynamics between academia and industry in driving innovation, and Dr. Dally's personal journey and motivations in the field of computer architecture.
Chapters
00:00:30 — Welcoming Dr. Bill Dally and What Excites Him Today
01:55 — The Future of Computing Innovation: Domain-Specific Accelerators in the Post-Moore Era
02:48 — The Core of Designing Domain-Specific Accelerators: A Programming Exercise
04:02 — Breaking the Cycle: Reformulating Algorithms for Hardware Co-Design
06:43 — Co-Design in Academia vs. Industry: Fostering Collaboration
07:58 — Nvidia's Approach to Innovation: Deep Learning, GPUs, and Ray Tracing
10:06 — Domain-Specific Accelerators for Legacy Compute and Emerging Applications
12:54 — The Critical Role of Interconnection Networks and Communication in Accelerators
14:42 — On-Chip vs. Off-Chip Networks: Theory, Cost Models, and Latency
22:37 — The Future of Domain-Specific Computing and Programming Paradigms
23:40 — The Mapping Problem: A Ripe Area for Research and Abstraction
29:39 — Pathways from Research to Product: Ensuring Impact and Successful Tech Transfer
38:15 — Dr. Bill Dally's Journey in Computer Architecture: From High School Dropout to Nvidia
45:46 — The Shift from Academia to Industry and the Importance of Impact
50:20 — Concerns About the "Brain Drain" from Academia to Industry
52:20 — Future Horizons: Generalizing Accelerators, Interconnection, and Design Processes
Takeaways
Domain-specific accelerators are crucial for future computing performance gains as Moore's Law slows, requiring a shift from general-purpose to specialized hardware.
Designing effective accelerators is fundamentally a programming exercise, focusing on understanding and reprogramming applications to be hardware-friendly and highly parallel, rather than just bolting on hardware to existing algorithms.
Successful technology transfer from research to product requires early and continuous collaboration between researchers and product teams, ensuring research addresses real-world constraints and product teams are invested in the new technology.
GPUs serve as a powerful platform for both developing and deploying domain-specific accelerators, allowing for experimentation with application co-design before committing to specialized hardware.
The incentives in academia and industry are currently misaligned, potentially hindering long-term innovation by drawing top talent away from academic research and teaching, which is vital for training the next generation of architects.