You can write your code in this way to avoid the unwanted grad calculation:
Q(s, a) = your_model(s, a)
with torch.no_grad():
    maxQ'(s',a') = your_model(s', a')

Then the forward calculation of maxQ will not accumulate grads on parameters of your model.