Hi,
I’ve been trying to train a GNN with pytorch. My sample graphs can have 1-8 nodes.
Unfortunately, the computation graph is too large to fit inside the resources I have. I split up the computation for each node in the graph network using a for loop that computes/propagates the messages, calculates the loss & calls backward. Inside this loop, I call a couple of functions from my model, not forward(). This helps me fit the whole thing inside a Titan with 12GBs of memory but I have 2 and the second one has remained unused.
I couldn’t use DataParallel or DistributedDataParallel because the wrapped models only have the forward function. I could technically fit everything in a single forward function and call it multiple times instead of once, but sample graphs do not have the same number of nodes. e.g. I can’t call losses.backward() 3 times in one process and 5 times in the other.
Is there any other way for me to utilize both GPUs?
I appreciate any tips or advice.