a forward pass with dummy inputs.
model instantiation with the torchscript flag.
These necessities imply several things developers should be careful about. These are detailed below.
Implications
TorchScript flag and tied weights
This flag is necessary because most of the language models in this repository have tied weights between their
Embedding layer and their Decoding layer. TorchScript does not allow the export of models that have tied
weights, therefore it is necessary to untie and clone the weights beforehand.
This implies that models instantiated with the torchscript flag have their Embedding layer and Decoding
layer separate, which means that they should not be trained down the line. Training would de-synchronize the two
layers, leading to unexpected results.
This is not the case for models that do not have a Language Model head, as those do not have tied weights. These models
