I think you just need to add padding. Given that your kernel size is 2x2 (even kernel-size),I think you can add manual padding only at one side of each dimension using torch.nn.ZeroPad2d
If you just use padding=1, that will increase the output size to 13x13:
>>> import torch
>>> import torch.nn.functional as F
>>> a = torch.randn(2048, 1, 12, 12)
>>> a.shape
torch.Size([2048, 1, 12, 12])
>>> kernel = torch.randn(2048, 1, 2, 2)
>>> b = F.conv2d(a, kernel, padding=1)
>>> b.shape
torch.Size([2048, 2048, 13, 13])

But if you manually add padding, your output will be the same shape as input:
>>> import torch.nn as nn
>>> pad = nn.ZeroPad2d((1, 0, 1, 0))
>>> ap = pad(a)
>>> ap.shape
torch.Size([2048, 1, 13, 13])
>>> b = F.conv2d(ap, kernel)
>>> b.shape
torch.Size([2048, 2048, 12, 12])