I guess it’s failing since the modules might not be easily wrapped inside an nn.Sequential container.
Running your code yields an error while creating the head module:
>  ModuleAttributeError: 'MemoryEfficientSwish' object has no attribute 'in_features'

which seems to indicate that the model.children() order might not reflect the actual execution order.
After removing the head module, I had to call the submodules of self.enc via:
    def forward(self, x):
        x = self.enc.extract_features(x)
        x = self.enc._avg_pooling(x)
        # x = self.head(x)      
        return x