Transformers, Pytorch, Machine Learning, Deep Dives, Data Science.
size) mask. From PyTorch docs.""" mask = (1 - torch.triu(torch.ones(size, size), diagonal=1)).bool() mask = mask.float().masked_fill(mask == 0, float('-inf')).masked_fill(mask == 1, float(0.0)) return mask Case study : a Word-Reverse Transformer Let’s now build our Transformer by bringing parts