Well, the problem it seems to come from the number of the classes (labels) or of  training file size. I diminished the size of the embeddings collected and the vocabulary  to half and still the same OOM. The only difference to the “successful” training situation is the number of labels and size of training file.
The OOM is currently occurring at stacking alphas (strangely) in the forward method.
Thanks