Or you could use my EfficientNet impl, it supports feature extraction out of the box 
I’m currently refining the feature, so most up to date is on my ‘features’ branch https://github.com/rwightman/pytorch-image-models/tree/features 11 .
It was intended for grabbing features appropriate for feature pyramid network style inputs (obj detection, segmentation, keypoints, etc) as you can see below… so it lets you grab the deepest set of features at each stride level at preset points. Not quite what you want as you seem to be keeping the last FC and have a more specific point in mind…
>>> import timm
>>> m = timm.create_model('efficientnet_b2', pretrained=True, features_only=True, out_indices=(1,4), feature_location='expansion)
>>> m.feature_info.module_name()
['blocks.1.2.conv_pwl', 'blocks.6.1.conv_pwl']
>>> m.feature_info.channels()
[144, 2112]

For your case though, you could utilise the utilities I’ve put together to help with feature extraction… see example below, skipping your projection/attn blocks for clarity
import timm
from timm.models.features import FeatureHooks

class AttentionNet(nn.Module):
    def __init__(self, num_classes):
        super().__init__()
        self.hook_loc = 'blocks.0.0.bn2'
        self.model = timm.create_model('efficientnet_b2', pretrained=True, num_classes=num_classes)
        self.hooks = FeatureHooks([dict(module=self.hook_loc)], self.model.named_modules())
        # ...

    def forward(self, x):
        x = self.model.forward(x)
        l1 = self.hooks.get_output(x.device)[self.hook_loc]

        # ....
        return [x, l1]

>>> aa = AttentionNet(1000)
>>> o = aa(torch.randn(2, 3, 224, 224))
>>> for x in o:
...   print(x.shape)
... 
torch.Size([2, 1000])
torch.Size([2, 16, 112, 112])