Thanks for the reply!
The shapes grow progressively from 10, 1024, 32, 32 to 10, 1024, 512, 512. And lut is of size 38.
I train with CUDA and I think that backprop is to blame here, CPU inference is comparable in my experience as well.