Hi again,
Thanks to let me understand your environment a little bit.
There is some occasions that CPU and GPU may have the same performance.
It depends the architecture of your network, may depend of the data volume are you using.
May be the Network is too shallow.
May be increasing the batch size of your GPU training you can see the difference compared to your CPU.
May the code have some latency between each batch sent to the GPU.
Without see your code/model and the way your data is trained is hard to have a clue of it.
Note: GPU accelerates matrix computations but if your computations are not so big or difficult at all to compute  almost the same performance can be achieved by the CPU. ( a modern GPU 1024+ threads while the top CPU today has only 200 threads).
Your instance has 4 cores that may give 8 threads, but to me it means your data is very easy to process that the CPU doesn’t have problems at all to do it.