Accurate and Efficient Fixed Point Inference for Deep Neural Networks

Deploying DNNs on embedded devices is a challenge because of their high memory and computational requirements. Performing DNN inference in lesser bit-width fixed point arithmetic is seen as a crucial step in realizing DNNs on embedded devices. State-of-the-art methods achieve floating point accuracy using re-training and complex activation normalization methods. In this paper we propose an accurate and efficient end-to-end DNN inference on 16-bit fixed point arithmetic. We prove that floating point accuracy can be achieved with a simple quantization method of using powers of 2 as scale factors coupled with our optimal bit-width estimation algorithm without using re-training. Additionally, it leads to efficient activation normalization using only arithmetic shifts. We show that the combination of our quantization method and activation normalization maximizes SIMD throughput resulting in 2x to 6x gain in execution time compared to floating point inference. Experimental results demonstrate that our method generalizes to different networks giving same or better accuracy compared to floating point for classification, regression and recurrent networks.