Sorry for the delay in my response. The results were bad until I normalized the batchnorm weights.
After this, I actually noticed that manually forward propping through the weights of my layers/batchnorm (since it already has normalized running_mean and running_var variables) yielded a higher accuracy than auto forward propping through the whole net.
Thanks for your help @ptrblck. Typically, I’ve only used batchnorm2d, and glorot initializations can be applied here, so didn’t think to apply any kind initialize on batchnorm1d.