XGBoosted_Learner:

weight_decay = 0.1


this is too high.
try 1e-5 or zero first
you cann’t use batch size 1 in train, if you are using batchnorm layer.
also many of optim methods need big batch size for good convergence.



 XGBoosted_Learner:

batch_size = 1


you should try simpler optim method like SGD first,try it with lr .05 and mumentum .9