Solved by Bahaa_Kattan in post #3 
              
              
                I played around with mnist dataset and reached to this code which is clean with more powerful accuracy, it converged faster then the above network so I think my problem solved. 
class Block(nn.Module):
    def __init__(self, input, output, p, kernel=3, padding=1, stride=1):
        super(Block, self…