We have the same exact issue, with the exception that we do not use a scheduler since we are using Adam. We also use L2 regularization via weight_decay parameter.  Did you find a solution @marcelodiaz @ptrblck?