The nn.Transformer 34 module by default uses 8 attention heads. Since the MultiHeadedAttention impl slices the model up into the number of head blocks (simply by a view operation) the model dimension must be a divisible by the number of heads. Please see also the documentation of nn.MultiheadAttention 300.