The evidence is flaky--a woman who has both confirmed and denied the story, corroborators with their own reasons to lie--but major scandals have been built on less.
One way to get even with someone is to fabricate a scandal about them.