Modulated Diffusion: Accelerating Generative Modeling with Modulated Quantization

Diffusion models have emerged as powerful gen-erative models, but their high computational cost in iterative sampling remains a significant bot-tleneck. In this work, we present an in-depth and insightful study of state-of-the-art accelera-tion techniques for diffusion models, including caching and quantization, and reveal their limita-tions in computation error and generation quality. To break these limits, this work introduces Modu-lated Diffusion (MoDiff), an innovative, rigorous, and principled framework that accelerates gener-ative modeling through modulated quantization and error compensation. MoDiff not only inherits the advantages of existing caching and quantiza-tion methods but also serves as a general frame-work to accelerate all diffusion models. The ad-vantages of MoDiff are supported by solid theoret-ical insight and analysis. In addition, extensive ex-periments on CIFAR-10 and LSUN demonstrate that MoDiff significantly reduces activation quan-tization from 8 bits to 3 bits without performance degradation in post-training quantization (PTQ). Our code implementation is available at https://github.com/WeizhiGao/MoDiff.