Why was the world of SAO not dominated by good real-life fencers? There are many people in real life who practice fencing for fun. Most of them (at least most of those around me) also play computer games about fighting. Sword Art Online clearly fits into the fighting genre of computer games. I didn't make a survey, but in any game about fencing it is certainly possible to find someone who likes do the same things in real life.

As I recall, in the world of SAO combos are quick, but very predictable, especially the strong ones, and their speed is not above human limits.

As I assume, if one real-life swordsman would get his hands (or his mind) on SAO, he would wreck everyone. One can move his weapon without combos, losing some speed, but getting a huge advantage of being able to use feints. Plus he would be able to stop his attacks to block enemy's blow, etc.

Yes, he would probably be weaker at the beginning due to not knowing the animations yet and most of combos being very short. Plus, he would probably need to get used to the pace of SAO, which may differ from real-life. But at higher levels he could both identify a combo by it's first "frames", plus quickly react. It would also be nearly impossible to hit him unless he doesn't expect an attack.

One could claim that he would lose muscle memory -- he wouldn't, since this process is more mental rather then some change applied to muscles.

Why did this not happen in the novel, why was it dominated by a bunch of Mary Sue-esque characters? Or did I miss something?
First and foremost, the amount of people actively playing the game was really low. From 10000 people at the start, 2000 died in first month and 3000-5000 stayed in starting city from fear of being killed. That leaves only some 3000 who actively played the game and of those only 300 were "on the front lines", with rest lagging behind.

Second, most people who bought the game were hardcore MMO players. Overlap between experienced kendo/fencing players and MMO players is quite small. So the probability of experienced swordsman being dragged into it is small, but not impossible, Kirito being prime example.

But even if such player was found, It wouldn't help him. From the anime adaptation, it might seem that SAO is almost-perfectly realistic simulation. That couldn't be far from truth. In novels, it is stressed multiple times that SAO's physics simulation is actually quite crappy. For example, player cannot just "put on" a coat or "grab" a weapon and liquids are pretty much an eye-candy with minimal physics interactions. I would argue same system would apply to sword fights. SAO sword fights would not be as flashy with people almost flying around. It would be mostly two guys standing still against each other and spamming sword skills and letting those do all the movements. And just hitting someone with sword is not that great if sword skill is not involved. So player's real-world sword experience would be practically useless in this scenario.