The man was swimming towards shore, not towards their boat.  The rest of those on the yacht didn't seem to have been alerted yet, or if they had, they were reacting silently.  For now at least, the team wouldn't have difficulty escaping the boat if they wanted to.