Hi,
The problem is that if all of the time is spent in the forward pass of your network, there isn’t much you can do without modifying this network or increasing your compute capabilities.

Using a lighter network (in term of compute, not memory) would be a solution: if you reduce the depth by 2, your runtime is going to be divided by 2. But depending on your application that might not be feasible.
Use a GPU instance as backend to speed up fw. Or use multicore cpu instances, that might help as well.
If you can’t use properly all the cpu cores you have, possibly batching requests could be possible (so that you can do batch forward passes)?

As a more general idea, you should be able from your model to compute roughly how many operations a single forward pass requires. That will give you an answer on how much hardware you will need to run this at the speed your want.