Make sure you are really replacing the last linear layer with your custom one, as different models might use different names for this layer (e.g. model.classifier vs. model.fc).
If that’s not the case, try to overfit a small data sample by playing around with some hyperparameters and scale up the problem once this is working.