I appreciate your help! Initially I was surprised by your answer, because shortly after I posted my question, I realized that the (more) correct way to do it is like this:
class Net(nn.Module):
	def __init__(self):
		super(Net, self).__init__()
		self.linear = nn.Linear(784, 10)

	def forward(self, x):
		return self.linear(x)

model = Net()
optimizer = torch.optim.SGD(model.parameters(), lr=0.001)
model.train()

for epoch in range(10):
	for i in range(100):

             orig_params = []
             for p in model.parameters():
	         orig_params.append(p.clone())
	         gaussian = Normal(loc=0, scale=torch.ones_like(p))
	         p.data = p.data + gaussian.sample()

	     output = model(input)
	     loss = nn.CrossEntropyLoss()(output, label)

             optimizer.zero_grad()
	     loss.backward()

             for p, orig_p in zip(model.parameters(), orig_params):
		 p.data = orig_p.data

	     optimizer.step()

However, now I see that the reason you didn’t see any issues with my initial example is that I simplified it too much. Because there’s no hidden layer to backpropagate through, it does not make any difference. However, if we consider, for example, MLP with a hidden layer:
class Net(nn.Module):
	def __init__(self):
		super(Net, self).__init__()
		self.linear1 = nn.Linear(784, 100)
                self.linear2 = nn.Linear(100, 10)
                self.relu = nn.ReLU()

	def forward(self, x):
		x = self.linear1(x)
                x = self.relu(x)
                x = self.linear2(x)
                return x

Well, now we would have a problem with the method in the first example: if we apply noise to both weight matrices (in linear1 and linear2), then the error would backpropagate through the original linear2 weights, while what we really want is to backpropagate through the distorted linear2 weights, to get the true gradient in respect to distorted weights of the linear1 layer. Do you agree?
By the way, I’m curious, how did you debug/verify the code?