Toward Efficient Low-Precision Training: Data Format Optimization and Hysteresis Quantization
As the complexity and size of deep neural networks continue to increase, low-precision training has been extensively studied in the last few years to reduce hardware overhead. Training performance is largely affected by the numeric formats representing different values in low-precision training, but finding an optimal format typically requires numerous training runs, which is a very time-consuming process. In this paper, we propose a method to efficiently find an optimal format without actual training of deep neural networks. We employ this method to determine an 8-bit format suitable for the training of various models. In addition, we propose a new quantization scheme that utilizes the hysteresis effect to suppress undesired weight fluctuation during training. This quantization scheme enables deeply quantized training using 4-bit weights, exhibiting only 0.2% degradation for ResNet-18 trained on ImageNet.
PDF Abstract
ImageNet
