Hi ptrblck,
Thank you so much for the reply. I post the complete stack trace here. It seems that bce_loss encounters invalid inputs.
After exploring any possible causes, I finally found that non-freezed parameter matrix will have nan value after ~600 iterations due to nan grad. I tested that manually set nan values to 0 if exist, which can successfully finish the training without error. My question is why grad of unfreezed trainable parameters can become nan after freezing certain parameters? Thanks.
update:
after adding torch.autograd.set_detect_anomaly(True), the stack trace indicates:
[W python_anomaly_mode.cpp:104] Warning: Error detected in PowBackward0. Traceback of forward call that caused the error:
File “/…/model/compgcn_conv.py”, line 81, in compute_norm
deg_inv		= deg.pow(-0.5)							# D^{-0.5}
(function _print_stack)
RuntimeError: Function ‘PowBackward0’ returned nan values in its 0th output.
the corresponding code is :
	def compute_norm(self, edge_index, num_ent, edge_weight=None):
		row, col	= edge_index
		edge_weight 	= torch.ones_like(row).float() if edge_weight is None else edge_weight 
		deg		= scatter_add( edge_weight, row, dim=0, dim_size=num_ent)	# Summing number of weights of the edges
		assert torch.isnan(deg).sum() == 0 # added for testing, this assert won't get triggered, meaning no nan in deg
		deg = deg.abs() # added for testing, forcing all positive, but doesn't work; however, "deg = deg.abs() + 1" works with no errors.
		deg_inv		= deg.pow(-0.5)							# D^{-0.5}
		deg_inv[deg_inv	== float('inf')] = 0
		norm		= deg_inv[row] * edge_weight * deg_inv[col]			# D^{-0.5}	

		return norm

and this happens at the first iteration not ~600, confused.