Did you experience the same thing with ShuffleNet v1?
ShuffleNet have splitting layers, which split the input to two parallel process.
Additionally, ShuffleNetv2 was design to be GPU efficient and ARM architecture. I wonder it would hurt the CPU speed.