Hi Yaroslav!
Here’s a little more information:
It turns out the the gradient for torch.sigmoid() is somewhat worse
than it could be.  Your SigmoidCustom displays similar imperfect
behavior, but is rather better than torch.sigmoid().
See this post:

  
    
    
    Imperfect gradient for torch.sigmoid() 
  
  
    Hi All! 
The gradient computation for torch.sigmoid() is not as good as it 
could or should be. 
Specifically, it underflows to zero sooner than it should. 
This can be seen by direct inspection of the gradient values.  Also, 
the gradient should be symmetric around zero, and this condition 
is violated.  Notably, the gradient underflows early for positive values 
of x. 
Also, the standard formula, sigmoid (x)  =  1 / (1 + exp (-x)), 
performs better, although still imperfectly, in that its grad…
  


Nonetheless, I think it is unlikely that this difference between your
SigmoidCustom and pytorch’s standard implementation explains
the issue you are seeing.  Both versions of sigmoid() are basically
okay, and the imperfection in their gradients is more of an edge case.
I would only think this would matter if your network were close to
training unstably.
I would suggest trying something like this:
class LinearNet(torch.nn.Module):
    def __init__(self, in_features,  out_features, hid_neurons=200, useCustom = True):
        super(LinearNet, self).__init__()
        if  useCustom:
            sig = SigmoidCustom()
        else:
            sig = torch.nn.Sigmoid()
        self.lin1 = LinearCustom(in_features, hid_neurons, bias=True)
        self.act1 = sig
        self.lin2 = LinearCustom(hid_neurons, hid_neurons, bias=True)
        self.act2 = sig
        self.lin3 = LinearCustom(hid_neurons, out_features, bias=True)
        self.softmax = torch.nn.Softmax(dim=1)

This way you can switch between the standard and custom sigmoid()
with a single flag and not risk letting some other bug creep in by changing
additional code.
Best.
K. Frank