Thanks for your response @googlebot . Identity check like c == d (or equivalently torch.eq(c, d)) is not the same as torch.equal(c, d), as torch.eq(c, d) is element-wise. However torch.eq(c, d).all() is equivalent to torch.equal(c, d). Below I test the torch.eq(c, d).all() alternative and it clearly shows that it does not suffer from these slowdowns!
New test
import copy
import timeit
import torch
from torch import nn

# Before self-attention (some computation)
c = torch.randn(1024, 2, 256, device='cuda')
d = torch.randn(1024, 2, 256, device='cuda')

t1 = timeit.timeit('torch.eq(c, d).all()', number=1, globals=globals())*1e3
t2 = timeit.timeit('torch.eq(c, d).all()', number=1, globals=globals())*1e3
t3 = timeit.timeit('torch.eq(c, d).all()', number=1, globals=globals())*1e3

print(f"First time with eq/all (before self-attention): {t1: .4f} ms")
print(f"Second time with eq/all (before self-attention): {t2: .4f} ms")
print(f"Third time with eq/all (before self-attention): {t3: .4f} ms\n")

t1 = timeit.timeit('torch.equal(c, d)', number=1, globals=globals())*1e3
t2 = timeit.timeit('torch.equal(c, d)', number=1, globals=globals())*1e3
t3 = timeit.timeit('torch.equal(c, d)', number=1, globals=globals())*1e3

print(f"First time with equal (before self-attention): {t1: .4f} ms")
print(f"Second time with equal (before self-attention): {t2: .4f} ms")
print(f"Third time with equal (before self-attention): {t3: .4f} ms\n")

# Perform self-attention (some computation)
a = torch.randn(1024, 2, 256, device='cuda')
b = torch.randn(1024, 2, 256, device='cuda')
_ = torch.matmul(a.transpose(1, 2), b)

self_attention = nn.MultiheadAttention(256, 8).to('cuda')
layers = nn.ModuleList([copy.deepcopy(self_attention) for _ in range(20)])

for layer in layers:
    a = layer(a+b, a+b, a)[0]

# After self-attention (some computation)
t1 = timeit.timeit('torch.eq(c, d).all()', number=1, globals=globals())*1e3
t2 = timeit.timeit('torch.eq(c, d).all()', number=1, globals=globals())*1e3
t3 = timeit.timeit('torch.eq(c, d).all()', number=1, globals=globals())*1e3

print(f"First time with eq/all (after self-attention): {t1: .4f} ms")
print(f"Second time with eq/all (after self-attention): {t2: .4f} ms")
print(f"Third time with eq/all (after self-attention): {t3: .4f} ms\n")

t1 = timeit.timeit('torch.equal(c, d)', number=1, globals=globals())*1e3
t2 = timeit.timeit('torch.equal(c, d)', number=1, globals=globals())*1e3
t3 = timeit.timeit('torch.equal(c, d)', number=1, globals=globals())*1e3

print(f"First time with equal (after self-attention): {t1: .4f} ms")
print(f"Second time with equal (after self-attention): {t2: .4f} ms")
print(f"Third time with equal (after self-attention): {t3: .4f} ms\n")

New results
First time with eq/all (before self-attention):  0.4065 ms
Second time with eq/all (before self-attention):  0.0777 ms
Third time with eq/all (before self-attention):  0.0763 ms

First time with equal (before self-attention):  0.1580 ms
Second time with equal (before self-attention):  0.1254 ms
Third time with equal (before self-attention):  0.1164 ms

First time with eq/all (after self-attention):  0.0825 ms
Second time with eq/all (after self-attention):  0.0775 ms
Third time with eq/all (after self-attention):  0.0776 ms

First time with equal (after self-attention):  4.1039 ms
Second time with equal (after self-attention):  0.1285 ms
Third time with equal (after self-attention):  0.1273 ms

Interpretation
It’s clear that the eq/all alternative does not suffer from this slowdown (first time after self-attention). I believe hence there must be a bug in torch.equal. I will create a corresponding issue on github.