Hi albanD,
Thanks for your suggestions. With regards to the forward passes are you referring to the training process?
In my case the training process has already been run on a separate, more powerful server with GPUs. So the checkpoint file is ready to use. Unfortunately the web app runs on a tiny web container instance with no GPUs available. I load the checkpoint into memory at startup, and it’s during the sampling (generate) process that I’m encountering performance issues.
The training was done on 3 layers, this proved more or less optimal - going down to 2 means the training loss is not as good. Do you mean that this can make a difference during the generate process as well?