That’s really interesting question. I’d also would like to know why.
As I can see not all models take twice as much memory on the card but Linear(1000, 1000) definitely does. Also it’s seen that with 3 cards available it takes the same amount of GPU memory, so it’s kind of invariant of the number of GPUs greater than 1