And he wasn't planning to waste his time with Naruto either. Also, those flashbacks you talk about are just his memories of Minato, a person he reminisces about because he was a close friend, and this was established in the arc Jiraiya is introduced in (and we knew Minato and Naruto were related since Minato's face was shown in the early Chuunin Exams).