Unfortunately, that is not the problem.
But I did a bit of memory usage tracking, by stepping through the main body of the training code, with all logging excluded, in a python shell. I found out that the significant memory usage increase happened at two lines of code, forward pass and backprop.
out = net(im, rois_h, rois_o, scores, ip)
...
loss.backward()

But the memory released after those two lines is always about 30MB smaller than the increase. So now it’s pretty clear that the memory leak happens either in my model, or the backprop. module. Given that no one has really complained about the backward() method, it’s highly likely I’ve done something wrong in the model forward pass.
Please take a look at the code
def forward(self, im, bh, bo, s, i):
# inputs are formatted in accordance to the three streams and the single neuron path
	"""
	Forward pass of the model. Note that 

	Inputs:
		im - input images
		bh - the human bounding box
		ho - the object bounding box
		s - classification score obtained from object detector
		i - the interaction pattern for an ho pair

	"""
	# compute features
	x = self.get_features(im)

	# human stream
	h = self.roipool(x, bh)
	h = h.view(-1, 256 * 6 * 6)
	h = F.dropout(F.relu(self.fc6_h(h)))
	h = self.fc8_h(F.dropout(F.relu(self.fc7_h(h))))

	# object stream
	o = self.roipool(x, bo)
	o = o.view(-1, 256 * 6 * 6)
	o = F.dropout(F.relu(self.fc6_o(o)))
	o = self.fc8_o(F.dropout(F.relu(self.fc7_o(o))))

	# pairwise stream
	i = self.maxpool(F.relu(self.conv1_p(i)))
	i = self.maxpool(F.relu(self.conv2_p(i)))
	i = i.view(-1, 32 * 12 * 12)
	i = self.fc4_p(F.relu(self.fc3_p(i)))
	
	# single neuron path
	s = self.fc2_s(self.fc1_s(s))
	
	# perform element-wise addition
	return (h + o + i + s)

def get_features(self, im):
	x = F.local_response_norm(self.maxpool(F.relu(self.conv1(im))), 5)
	x = F.local_response_norm(self.maxpool(F.relu(self.conv2(x))), 5)
	x = F.relu(self.conv3(x))
	x = F.relu(self.conv4(x))
	x = F.relu(self.conv5(x))
	return x