Based on both of your provided errors this check if isinstance(output, torch.Tensor): doesn’t work because output has type list. I don’t know how this supposed to work, but output coming from non_max_suppression function in test.py is a list. And then you check non_max_suppression function its supposed return is also a list and they are never converting that to tensor somehow.
You should probably dig dipper in the code and rewrite it a bit saving the logic. You may want to move outputs of the model to cpu before non_max_suppression call in test.py or do the same in output_to_target on the output variable iteratively