Awesome. I worked it out and it ended up being the matrix you get from keeping every other D rows of an identity matrix of size L. Here’s an implementation and visualization.
import torch
import matplotlib.pyplot as plt

kernel_size = 7
dilation = 3

x = torch.rand(kernel_size, kernel_size)
eff_kernel_size = kernel_size + (kernel_size-1)*(dilation-1)

h = torch.eye(eff_kernel_size)[::dilation]
d = torch.mm(torch.mm(x.t(), h).t(), h)

fig = plt.figure()
ax1 = fig.add_subplot(1,2,1)
ax1.matshow(x.numpy(), interpolation='nearest')
ax1.title.set_text('Dense')
ax2 = fig.add_subplot(1,2,2)
ax2.matshow(d.numpy(), interpolation='nearest')
ax2.title.set_text('Dilated')
plt.show()


EDIT: Since PyTorch 1.9, there is torch.kron() which performs the Kronecker product. So you can get a pretty good speed-up by simply running:
h = torch.zeros(dilation, dilation)
h[0,0] = 1
d = torch.kron(x, h)[:eff_kernel_size, :eff_kernel_size]


dilation910×538 10.3 KB