Nice, thanks! It works now.
For any interested:
I tried the in place variant too, but without the no_grad context. That gave me an error that in place is not allowed.
And by the way, I had to zero the gradient after each step:
        self.m.grad.data.zero_()
        self.b.grad.data.zero_()

Anyone have any idea why the context manager is called no_grad(). It sounds like no gradients are begin computed, but what I understand from you it makes sure the graph is not recalculated.
I find the naming in ML libraries I encountered pretty cryptic at times, but that might be caused by me being new to this