SMoLPU: An Energy-Efficient Sparse Mixture-of-Experts-Based Speculative Decoding Language Processing Unit https://sscs.ieee.org/wp-content/themes/movedo/images/empty/thumbnail.jpg 150 150 https://secure.gravatar.com/avatar/8fcdccb598784519a6037b6f80b02dee03caa773fc8d223c13bfce179d70f915?s=96&d=mm&r=g
Abstract:
Reasoning strategies such as chain-of-thought improve the accuracy of large language models (LLMs) but lengthen decoding sequences, inflating external memory access (EMA) on mobile devices. Numerous optimizations have been explored to reduce EMA, from conventional techniques such as group quantization and channel-wise pruning to sparse mixture-of-experts (MoE) and speculative decoding (…