zhuser:

The question is when I want to use a layer three times (such as the case above) can I initialise it as one layer and repeat it in the forward method several times (here three) or do I have to initialise a layer as many times as I want to use that layer.


Both approaches are valid and it depends, if you want to reuse the parameters (and layers).
If you initialize a layer once, it’ll have one set of parameters, which will be optimized. In the forward method you can reuse this layer as many times as you want as seen here in a simple example:
lin = nn.Linear(1, 1, bias=False)
optimizer = torch.optim.SGD(lin.parameters(), lr=1.)

x = torch.randn(1, 1)

# single step
out = lin(x)
out.backward()
print(lin.weight.grad)
optimizer.step()
lin.zero_grad()

# reuse the layer
out = lin(x)
for _ in range(10):
    out = lin(out)
out.backward()
print(lin.weight.grad)
optimizer.step()
lin.zero_grad()

This “reuse” of the layer is sometimes also used and called “parameter sharing”. E.g. you could initialize specific layers in an encoder and “share” them in the decoder, which would mean that you either directly reuse them in the decoder path or reshape the parameters somehow so that they can be used in another part of the model.
I’m not familiar with your use case so don’t know which approach would be the right one for your model.
However, be a bit careful in reusing batchnorm layers, as they are tracking the running stats by default and I’ve seen some users reporting that reusing these layers is often failing due to this.