Transformers, Pytorch, Machine Learning, Deep Dives, Data Science.
Final dim will be (batch_size, num_heads, seq_length, , hidden_dim // num_heads) def combine_heads(self, x): batch_size, num_heads, seq_length, head_hidden_dim = x.size() return x.transpose(1, 2).contiguous().view(batch_size, seq_length, num_heads * head_hidden_dim) def forward( self, q, k, v,