SQProp is an algorithm to compute forward and backward propagation with mixed-precision numbers.

Reducing numerical precision is critical to achieve fast and economic development and deployment of deep neural networks. While most existing reduced-precision algorithms focus only on the forward propagation, SQProp provides an unified framework for both forward and back propagation, making it suitable for accelerating both the training and inference of deep neural networks.

SQProp is based on Stochastic Quantization with advanced variance reduction technique. Instead of the worth-case analysis of error accumulation, SQProp comes with a statistical framework that analyzes the bias and variance of the gradient. This enables the development of unbiased, variance-reduced gradient estimators. The theory and implementation of SQProp are still under active development. Initial results show some promise on training the entire network with a mix of unprecedentedly low 4-bit integers and 32-bit floating point numbers.