Thank you very much again for your help @ptrblck, really, really appreciate it!  Very sorry regarding the images - my apologies!
I couldn’t reproduce the issue with synthetic data, but if I modify the example that you shared to work with my data, the issue appears.
I saved the data as follows (I uploaded it here 1, it is under 8MB, forums say I cannot upload it here):
it = iter(train_dl)
x1, t1 = next(it)
x2, t2 = next(it)

torch.save([[x1, t1], [x2, t2]], 'data.pth')

This is the code I run
data = torch.load('data.pth')

class FrontEnd(nn.Module):
    def __init__(self):
        super().__init__()
        self.register_parameter('alpha', torch.nn.Parameter(torch.tensor(0.)))
        
    def forward(self, x):
        x = x ** torch.sigmoid(self.alpha) # <----- line causing issues
        return x

class Model(nn.Module):
    def __init__(self):
        super().__init__()
        self.frontend = FrontEnd()
        self.lin = nn.Linear(10, 264)
    
    def forward(self, x):
        x = self.frontend(x)
        x = x.mean((2,3,4))
        return self.lin(x)
    
model = Model()
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)
criterion = nn.BCEWithLogitsLoss()

for x, target in data:
    optimizer.zero_grad()
    out = model(x)
    print(torch.isfinite(out).all())
    if not torch.isfinite(out).all(): break
    loss = criterion(out, target)
    loss.backward()
    optimizer.step()

And this is the output I receive
tensor(True)
tensor(False)

The author of the paper implemented this operation using lasagne and theano - my data is different but I should be preprocessing it how he does in the paper.