A Novel Decay Step Size For Stochastic Gradient Descent

In this paper, we propose a modified version of the $frac{1}{sqrt{t}}$ step size and demonstrate its convergence rate of $O(frac{ln T}{sqrt{T}})$ for smooth non-convex functions without the Polyak-Łojasiewicz condition. Through experiments on the FashionMNIST, CIFAR10, and CIFAR100 datasets, we show that the modified step size significantly improves both test accuracy and training loss in comparison to the original $frac{1}{sqrt{t}}$ step size.