I faced similar problem while using pretrained EfficientNet. The issue is with all variants of EfficientNet, when you install from pip install efficientnet-pytorch.
When you load the pretrained network, set include_top as False, and modify classifier (tail part)
EfficientNet.from_pretrained(‘efficientnet-b7’, include_top = False)
Last 3 children (dropout, fc and swish) of model will not be considered now.
Hope it helps!