The evidence is flaky--a woman who has both confirmed and denied the story, corroborators with their own reasons to lie--but major scandals have been built on less.
Scandals don't have to be based on truth.