Solved by ptrblck in post #2 
              
              
                By dead ReLUs you mean a negative input and thus a zero result? 
If so, you could count the zeros in the output activation. 
Here is a toy example for a simple model: 
model = nn.Sequential(
    nn.Linear(10, 10),
    nn.ReLU()
)

x = torch.randn(10, 10)
output = model(x)
(output == 0).sum(1).float(…