Transformers, Pytorch, Machine Learning, Deep Dives, Data Science.
float('-inf')).masked_fill(mask == 1, float(0.0)) return mask def encode( self, x: torch.Tensor, ) -> torch.Tensor: """ Input x: (B, S) with elements in (0, C) where C is num_classes Output (B, S, E) embedding """ mask = (x == self.PAD_IDX).float() encoder_padding_mask = mask.masked_fill(mask == 1,