I would recommend to check out Tim Dettmers’ great posts about choosing GPUs for deep learning 15 and Hardware Guide to Deep Learning 6. The guides do not focus on distributed setup, but there are suggestions on multiGPU machines and how to select a GPU for your task and budget.