Think of a trivial model such as y=mx+b where x is the input, y the output, and m and b are the weights and biases.
Ignoring PyTorch, you could implement a model as something like:
class Model():
   def __init__(self, weight, bias):
       self.weight=weight
       self.bias=bias
    def forward(self, input):
        return input * self.weight + self.bias

This would be “stateful” because the weights and biases are member variables, part of the “state” of the model class.
Alternatively you could write:
class Model():
    def forward(self, input, weight, bias):
        return input * weight + bias

This would be stateless, the model is parameterized. Any updates to the model parameters happen outside the model.
Moving from the former example to the second example is described in the paper as:

Here, it replaces the call to the forward method with one which first replaces the stateful parameters of the submodule with ones provided as additional arguments to the patched forward, before calling the original class’s bound forward method, which will now used the parameters provided at call time.

You can see this is an equivalent migration to the above example.
You may find information about stateful vs stateless LSTMs or RNNs when searching for more info, but that refers to a different kind of state from what’s referred to in this paper.