Why repeat initializing loss modules in every forward?
Hello,

I find that your implementations usually initialize loss modules, e.g., nn.CrossEntropyLoss, inside models' forward functions. I am curious about the reason of doing this. Generally, in Pytorch, a module should be initialized in __init__ and used in forward. Does the frequent initialization cause overhead and memory issues? 

Thanks,
The module `nn.CrossEntropyLoss` does not contain any weights so here we don't allocate any memory really when initializing the module at every forward. However if we would do this with `nn.Linear(...)` at every forward step it should be considered bad practice IMO since in this case we would allocate a big tensor (the weights of the linear layer) at every forward step.<|||||>@patrickvonplaten , thanks for the answer. Pytorch has `functional.cross_entropy()`, which should be more suitable to use in forward. Although `nn.CrossEntropyLoss ` doesn't cause overhead, it doesn't follow Pytorch's convention that initializes a module in init and use it in forward. It confused me a little lit when reading the code. I was wondering any specific reason of using a `nn` module instead of a `functional` method in forward.<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.