Abstract:
Conventional large language models (LLMs) have large parameter sets, making inference costly and energy-intensive; Mixture-of-Experts (MoE) mitigates this by activating fewer weights per token, but fine-grained MoE LLMs still face runtime workload variability, inefficient conventional scheduling, and high HBM loading energy/bandwidth demands. A3D-MoE addresses these with 3D heterogeneous …