### Comment:
Here's a problem with google TPUs - sometimes most of your deep learning code runs great on the tensor chip but in every training loop you have a small amount of computation the TPU can't handle. Now you have to transfer all your crap off the chip onto a traditional server to do that step, then transfer it back to train the next batch. In this process, the bottleneck is actually the data transfer. They're working on this, but it's a real problem.

### REPLY:
Its why nvidia bought mellanox.

### END.