It looks like your batches are too small.  If your batch size is 128, that means the 2 gpu case has a batch of 64 per GPU.  Also your profile shows almost all the time in cuda memory allocation.
Try making the batch as large as you can, then see how things behave.