Just reread your post.  

Yes I would visit his office often (at least 3 days a week) and we would have long conversations with the door closed.  He shared some of his past traumas with me.  

From experience, people can really bond over sharing past trauma, so if they've been through something similar and they can help each other see it from a new perspective, that's HUGE.

You mentioned they had dinner together.  If it was just once I wouldn't be that concerned.