Deep Learning, Sora, Computer Vision, Transformers, Attention.
value(pout) # scale_factor = 1 / math.sqrt(q.size(-1)) # qk = q@k.transpose(-2, -1) * scale_factor #(B, T, 16) @ (B, 16, T) --. B, T, T # qk.shape q.shape, k.shape, v.shape (torch.Size([4, 480, 768]), torch.Size([4, 480, 768]), torch.Size([4, 480, 768])) since q and k are of shape (4, 480, 768) x