I would recommend to add print statements for the current tensor shape in the forward methods in all custom modules to narrow down where the shape mismatch is coming from.
Your current code snippets uses another undefined model (ProposalNet) so that I cannot debug is myself.