Iterative procedures for parameter estimation based on stochastic gradient descent (sgd) allow the estimation to scale to massive data sets. However, they typically suer from numerical instability, while estimators based on sgd are statistically inecient as they do not use all the informationinthedataset. Toaddressthesetwoissueswe propose an iterative estimation procedure termed averaged implicit sgd (ai-sgd). For statistical eciency ai-sgd employs averaging of the iterates, which achieves the Cramer-Rao bound under strong convexity, i.e., it is asymptotically an optimal unbiased estimator of the true parameter value. For numerical stability ai-sgd employs animplicitupdateateachiteration, whichissimilar to updates performed by proximal operators in optimization. In practice, ai-sgd achieves competitive performance with state-of-the-art procedures. Furthermore, it is more stable than averaging procedures that do not employ proximal updates, and is simple to implement as it requires fewer tunable hyperparameters than procedures that do employ proximal updates.
['Panos Toulis', 'Dustin Tran', 'Edoardo M. Airoldi']
Towards stability and optimality in stochastic gradient descent