Transformers, Pytorch, Machine Learning, Deep Dives, Data Science.
1::2] = torch.cos(position * div_term) pe = pe.unsqueeze(0) self.register_buffer('pe', pe) def forward(self, x): """ Arguments: x: Tensor, shape ``[batch_size, seq_len, embedding_dim]`` """ x = x + self.pe[:, :x.size(1), :] return x Encoders We are getting close to having a full encoder working !