Did you try out the approach of splitting the model on GPU and CPU?
I’m not sure about replacing the linear layers with average pooling.
Does the model performance stay the same more or less?
What did you observe?