How did you use pretrained=True with 4 input channels?
Did you copy the pretrained 3 channels to your newly initialized first conv layer with 4 input channels?
Training a model from scratch is expected to take more time then using a pretrained model for most use cases.