Hi Taher!
There are several issues here.  See my in-line comments, below.



 Taher_Anjary:

The task is to  perform binary classification on Ocular Disease Recognition, ODIR5k, dataset.


For binary classification, you should be using BCEWithLogitsLoss
and get rid of the sigmoid() at the end of your model.  (You could
also keep the sigmoid() and use BCELoss, but with greater numerical
instability.)  MSELoss isn’t the right choice for binary classification.


Images are 128x256 rgb images. I process them into grayscale


As an aside, your rgb images have information in the color.  So it
would make sense not to convert them to grayscale.  But to make
use of the color information, you would have to create a model
that did make use of the color – typically by having it accept inputs
with a channel dimension for the three color channels.


        y = torch.sigmoid(self.linear3(z2))



As mentioned above, get rid of the sigmoid().


    criterion = torch.nn.MSELoss(reduction='mean')



As mentioned above, you should use BCEWithLogitsLoss.


        labels = labels.float() #.cuda()



Your labels should, indeed, be floats for use with BCEWithLogitsLoss.
By the way, what type were they before you called float()?
Also, if you continue to have problems, you should check what values
they take on.  They most likely take on values of 0.0 (the “class label”
for class-“0”) and 1.0 (for class-“1”).  But they could be values
(understood to be probabilities) that range over [0.0, 1.0].  In any
event, whether discrete or continuous, they should be floats.


        y_2 = torch.zeros(len(y_pred))
        y_2[y_pred>=0.5] = 1
        y_2 = y_2.int()
        correct = 1*(y_2 == labels.int()).sum().item()



This is a reasonable way to convert y_pred (the output of your model)
to yes-no (0-1) predictions, suitable for computing your accuracy.
Note, this is true if y_pred in made up of probabilities that range
over [0.0, 1.0], as will be the case if you have the final sigmoid()
in your model.  However, if you get rid of the sigmoid() and use
BCEWithLogitsLoss (as you should), your outputs will be raw-score
logits that range over (-inf, inf), and you should use a threshold
of 0.0, rather than 0.5, that is:
# y_2[y_pred>=0.5] = 1   # change this
y_2[y_pred>=0.0] = 1     # to this

Also, your accuracy code it somewhat roundabout, and could be
tightened up.  But that is left as an exercise for the reader …


        correct = (y_pred == labels).sum().item()



This, however, is a problem.  After calculating correct in a sensible
way, you immediately overwrite it with the exact-equality-test version
that is incorrect.  So your (train) accuracy will be wrong in the same
way as it was in your first version.


        accuracy = correct / total



I see that you’ve gotten rid of the factor of 100 you had in your first
version.  I suppose that you are now calculating your accuracy as a
“fraction” where it had been a “percentage” before.  No matter, but it
makes comparing this version with the first more confusing.


            y_pred = y_pred.flatten()
            y2 = torch.zeros(len(y_pred))
            y2[y_pred>=0.5] = 1
            y2 = y2.int()
            total = labels.size(0)
            correct = (y2 == labels.int()).sum().item()
            acc = correct / total



As an aside, in test() you are calculating accuracy correctly, and
not making the exact-equality-test mistake you kept in train().
This explains the difference between your training accuracy and
validation accuracy in your results below.


In any case, running the code produces the following output:
…
As you may observe, nothing  is changing.


Redo this with BCEWithLogitsLoss and the correct accuracy
calculation.


I do suspect I may not be using zero_grad correctly but I’m really not sure.


Your use of optimizer.zero_grad() is correct.


I have tried several learning rates: 0.1, 0.01, 0.00, 0.0000001


After you get your code more-or-less working, it does make sense
to try different learning rates.  Note, that a learning rate of 0.00
doesn’t make sense – it multiplies the gradient by 0.00, in effect,
doing nothing.  Also, 0.0000001 seems  awfully small.


and diffrerent batch sizes: 4, 5, 10, 25


And experimenting with different batch sizes makes sense, as well.
Good luck.
K. Frank