You could probably use the groups parameter:
conv_x = nn.Conv2d(512, 512, kernel_size=3, stride=1, padding=1, bias=False, groups=512)
conv_x.weight = nn.Parameter(torch.from_numpy(grad_x_weights).float().expand(512, 1, 3,3))

grad_x = conv_x(X)

To freeze the the weights you could set their requires_grad flag to False.