Payne, 27, who needed a top-10 finish to qualify for Rio, clocked one hour 58 minutes and 58.53 seconds in Kazan.
Aurelie Muller of France finished 49.3 secs faster to take gold, ahead of Dutchwoman Sharon Van Rouwendaal and Brazil's Ana Chunha, who was third.
"It's not the result I wanted, but it's not the end," Payne told BBC Sport.
The two-time world champion and team-mate Danielle Huskisson, who was 35th, will have a second opportunity to qualify for Rio at the Olympic qualification event in Portugal in June next year.
Fellow Briton Jack Burnell secured his place in Rio by finishing fifth in the men's event on Monday.
Payne, the 2008 Beijing Olympic silver medallist, had been undecided about whether to continue in the sport after finishing fourth at London 2012.
"I've only been back in full-time training for a year and maybe I've raced too much or too little," she told BBC Sport.
"I'm technically old at 27 in the sport, but am still learning and trying out new things all of the time so I think there's a lot to build on from here."
Huskisson will battle Payne for the sole position Britain can now attain for the women's 10km race at the Rio Olympics and believes officials can do more to ensure fairer competition.
"Most of the girls seem to think it's rugby rather than open water swimming," said Huskisson, who received a yellow card whilst trying to maintain position.
"There are definitely favoured swimmers, but there's nothing you can do but go out and swim as hard as you can, which I'll continue to do."
Keri-Anne Payne missed out an automatic place at the 2016 Olympics, finishing 15th in the World Championship 10km marathon swimming event in Russia.