You are not missing anything and your explanation seems fine.
As @bdh said, it’s comparable to a conv or pooling layer.
Here is a small example for your use case:
image = torch.randn(1, 1, 5, 5)

kh, kw = 3, 3 # kernel size
dh, dw = 1, 1 # stride
patches = image.unfold(2, kh, dh).unfold(3, kw, dw)
patches = patches.contiguous().view(-1, 9)

lin = nn.Linear(9, 1)
output = lin(patches)
output.shape
output = output.view(1, 1, 3, 3)