As you can find in various source on how to debug CNN and training:

try to overfit the model on a single batch

Seems like this model without BN has a difficulty even to overfit a single batch.
I tried a larger LR -> 0.1 and loss on a single batch stucked on ~4.0
With BN’s seems like it can overfit a batch and can train (as you also observed).
If you want to remove BN’s, probably you need to check conv weights initialization schema and find out an appropriate one.
It seems like you have some bugs:

input data is 40x40

train_transform = Compose([
    RandomCrop(32),
    RandomHorizontalFlip(),
    Pad(4),
    ToTensor(),
    Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]),
    #LinearTransformation(transformation_matrix, mean_vector), 
 ])

usually, they do pad first and than random crop of 32

data is normalized -> do not need to divide by 255 in whitening

def ZCA_whitening(data):
    
    data=data.view(data.size(0),-1)
    X_norm=data/255  # <--- ???

HTH