That’s a great question. Unfortunately, I don’t have any good answer to that, but I would be very interested to hear whether @ptrblck 's suggestion, using SGD, addresses the problem in you case. (And in case you find another good solution in future, please share!).