Mageean's time of 2:00.79 cut 2.05 seconds off her previous best but she was passed in the closing 15 metres by England's Alex Bell who won in 2:00.53.
The Portaferry woman had been tipped to challenge Rose-Anne Galligan's Irish record of 2:00.58.
Pacemaker Sinead Denny went to the bell in 59.45 with Mageean just in arrears.
When European Championship 400m semi-finalist Denny dropped out after 450 metres, it left Mageean with a two-metre lead but Bell, a quality athlete who just missed on Olympic qualification, stayed in touch before unleashing her late run to pass the Northern Irishwoman.
Yorkshire athlete Bell's winning time cut 0.76 seconds off her previous personal best.
After her first run on home soil since winning her European bronze medal two weeks ago, Mageean made no attempt to hide her frustration after being pipped by the 23-year-old.
"I'm very disappointed. I had to do a lot of the work in that race and just got run down in the last few metres," said Mageean, who will head to the Ireland pre-Olympic training camp in Rio early next week.
"I'll be back to win the 800m and the mile at the Morton Games in the future."
There were American winners in both mile races with Katie Mackey setting a new women's meeting record of 4:25.48 and John Gregorek also improving his personal best as he pipped compatriot Kyle Merber to win the men's event in 3:55.57.
The quality of the race was such that even 11th-placed Irishman Paul Robinson, back in action after a long injury spell, broke four minutes as he clocked 3:58.75.
The men's 400m produced the tightest of finishes with Sudanese athlete Sadam Koumi given the verdict as he was given the same time as Guyana's Winston George.
With Britain's Jared Dunn (46.28) in third, Brian Gregan finished fourth in 46.59 to edge out David Gillick (46.93) in the battle of the Irish athletes.
There was a predictable American one-two in the men's 100m as US Olympic trials semi-finals Joe Morris (10.27) pipped Dentarius Locke (10.27) who finished eighth in the final in Eugene.
With Amy Foster, like Jason Smyth, a late withdrawal from the meeting, Nigeria's Stephani Kalu won the women's 100m in 11.60 which left her ahead of Irish pair Niamh Whelan (11.72) and Joan Healy (11.73).
Meanwhile, Mark English showed encouraging form as he battles for form after injury as he clocked a big season's best of 1:45.36 at the Diamond League meeting at London's Olympic Stadium.
English, who only recently return to action after a stress fracture, was 1.48 seconds behind French winner Pierre-Ambroise Bosse but it was still a very encouraging run for the Donegal man ahead of the Olympics.

A Derry board statement said Barton had been appointed in September 2015 "for an agreed period of two years".
"We would like to sincerely thank Damian, his management and all players who have represented their county during this period," it read.
The statement added that nominations for the job would be sought from clubs.
Barton had a creditable first year in charge of Derry as his side reached the Dr McKenna Cup Final and also narrowly missed out on qualifying for the All-Ireland quarter-finals after losing by a point to eventual semi-finalists Tipperary in the last qualifying round.
However despite that promising opening season, Barton's squad was badly depleted this year by the withdrawal of several experienced campaigners.
The under-strength Oak Leafers suffered relegation to Division Three of the Football League on points difference and then were well beaten by Tyrone in their opening Ulster SFC game for the second successive season.
While they regrouped to beat Waterford in the qualifiers, Saturday's extra-time defeat by Mayo saw Derry exiting from the championship.
Barton side were leading the Castlebar game up until a minute from the end of normal time but despite a brave effort, Derry had nothing left in extra time as Mayo eventually won 2-21 to 1-13.
European medallist Ciara Mageean was left "disappointed" as she was edged out over 800m at the Morton Games despite setting a big personal best.
[NEXT_CONCEPT]
Damian Barton's reign as Derry football manager appears to be over after the county's GAA board asked for nominations for the post.