Hi Pytorch,
Are there best practices for avoiding out of memory errors when using dataparallel. For instance, I have 8 cards, but I think the gradients are all accumulated on one card (card 0 by default), so would it make sense to use 7 cards, and set the 8th one as the output device? Or would that be a waste? I’ve tried experimenting with it but the overhead is so high I’m not sure i’d get there quickly.