Certainly, Subaru's statement had been very frivolous, but Emilia brushed it off like it had nothing to do with her. The fact that she clearly didn't see herself as the target of Subaru's affections gave him no route to follow up.