Hi Hamad!
First, I think going through an exercise like this is very worthwhile.
Taking the time to do “by hand” some of the things built in to pytorch
is a great way to really learn what is going on and will prove valuable
when you move on to tackling more complicated problems.



 msee19018:

I want to make a simple classification neural network with pytorch’s autograd package.


I have not gone through your code in detail, so there may be other
issues, but I do see a couple of things that you should look into.


class network():
def __init__(self,num_layer,layer_size ): 

…
    self.act=torch.nn.ReLU()



As a minor note, you might want to remove or comment out things
such as self.act that are unused.  Doing so could make your code
more readable, and perhaps less error-prone.


    self.act1=torch.nn.Sigmoid()

…
def forward(self,x):

    return self.act1(torch.matmul(self.act1(torch.matmul(self.act1(torch.matmul(self.act1(torch.matmul(x,self.W1)+self.b1),self.W2) +self.b2),self.W3) +self.b3),self.W4)+self.b4)



Applying a Sigmoid layer to your final linear layer is not correct for
a multi-class classification problem where you use cross-entropy as
your loss.  Doing so will convert each of your ten outputs into an
independent probability (between 0.0 and 1.0), but you want a
probability distribution over the ten classes.  That is, you want a set
of ten values that are not only individually between 0.0 and 1.0,
but that sum to 1.0 as well.
To get such a probability distribution you want your final activation
layer to be your self.act_last=torch.nn.Softmax(dim=1) .
(However, what you really want is not to have a final activation layer,
to work directly with the raw-score logits generated by your final linear,
and to write a “cross-entropy-with-logits” loss function that uses
log_softmax() internally.)


def loss(self,pred,Y):

    return -1*(torch.sum(Y*(torch.log(pred))))/20



You don’t say what Y is here, but, as written, Y should be one-hot
encoded class labels (or more generally, “soft labels” that are given
by a probability distribution over the classes – each in [0.0, 1.0],
and that sum to 1.0).


model=network(5,[784,392,196,98,10])
lr=0.00001
n_iters=100


Some side comments:
As it currently stands, your network has what we call three “hidden”
layers.  I would also encourage you to experiment with “shallower”
networks with only one or two hidden layers.
Your learning rate of 0.00001 strikes me as being smaller than it
needs to be.  You might be able to train faster with a larger learning
rate (although it is sometimes necessary to start with a smaller rate
during a “warm-up” period and it can be helpful to reduce the rate
towards the end of your training to avoid jumping back and forth
across “gullies.”)
You will almost certainly need more that 100 epochs.


    with torch.no_grad():

        model.W1-=lr*model.W1.grad

        model.W2-=lr*model.W2.grad

        model.W3-=lr*model.W3.grad

    # value_W4=self.W4-lr*self.W4.grad



Now, the big one:  You aren’t optimizing your final linear layer.  This
will make it very hard for your network to train, as the upstream
layers will have to “learn” to undo the damage being done by your
randomly-initialized – and frozen – final layer.
Good luck.
K. Frank