How big is the network? You’ll get a much bigger speedup for a larger network.
Also - what type of network is it? How is it implemented? I assume it’s running just fine (no errors being thrown)? Are you timing forward passes or training?