That’s a bit weird. I cannot reproduce this issue using this code snippet:
class FrontEnd(nn.Module):
    def __init__(self):
        super().__init__()
        self.register_parameter('alpha', torch.nn.Parameter(torch.tensor(0.)))
        
    def forward(self, x):
        x = x ** torch.sigmoid(self.alpha) # <----- line causing issues
        return x

class Model(nn.Module):
    def __init__(self):
        super().__init__()
        self.frontend = FrontEnd()
        self.lin = nn.Linear(10, 2)
    
    def forward(self, x):
        x = self.frontend(x)
        return self.lin(x)


model = Model()
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)
criterion = nn.CrossEntropyLoss()

x = torch.empty(10, 10).uniform_(0, 10)
target = torch.randint(0, 2, (10,))

for i in range(100):
    optimizer.zero_grad()
    out = model(x)
    print(torch.isfinite(out).all())
    loss = criterion(out, target)
    loss.backward()
    optimizer.step()

I reduced the number of neurons a bit, but that shouldn’t make a difference.
Could you run my code and check, if you are still getting NaNs?
Also, do you see any difference between my and your code?
PS: It’s better to post code directly by wrapping it into three backticks ```, as I could simply copy-paste it