Simons blamed construction costs and archaeology requirements for its decision to ask West Dorset District Council to find alternative developers.
West Dorset District Council had agreed funding of up to Â£4m towards the scheme which includes Marks & Spencer and Waitrose stores.
The council said it was "disappointed" by the decision.
The company cited the requirements of English Heritage, including undertaking a full archaeological survey, as making the project financially unviable.
In a letter to the council, Simons' chief executive Tom Robinson said the viability of the second phase of the project had been "marginal for a number of years".
An online petition set up by residents' group Defend our Rich Cultural Heritage backing calls for archaeological investigations of the site, which lies within the walls of the old Roman town, has received almost 4,000 signatures.
Anthony Alford, leader of West Dorset District Council, said: "We remain ambitious for this key town centre site.
"Our vision is to keep Dorchester a thriving shopping destination and keep retailers in our county town."
New council buildings, a library and an adult learning centre have already been built in Charles Street in the first phase of its regeneration.

A first-half David McAllister own goal was all that separated the sides in a tight affair at Mourneview Park.
Coleraine hammered Glentoran 4-1 at the Showgrounds with James McLaughlin scoring twice.
David Jeffrey's Ballymena United ran out easy 4-1 winners away to Carrick Rangers to move into the top half of the table.
Relive the action from Friday's Premiership matches
Despite suffering their first defeat of the season, Ards are still tied on points at the top of the table.
There were few clear chances in the game, with defences on top before McAllister nodded past keeper Aaron Hogg and into his own net after 17 minutes of play.
Crusaders and Linfield can overtake the joint leaders if they win their respective games on Saturday.
The Bannsiders put four goals past Glentoran, who continued their disappointing start to the league campaign.
Coleraine took the lead after eight minutes when referee Mervyn Smith pointed to the penalty spot when Elliott Morris was adjudged to have brought down James McLaughlin despite getting a strong palm on the ball.
Neil McCafferty made no mistake from the spot by slotting past Morris, although the Glentoran keeper may feel aggrieved that he wasn't ready to face the kick.
Three goals in nine minutes assured Coleraine of the points. Darren McCauley tucked home after being played through by Rodney Brown, before McLaughlin drove the ball home from the edge of the penalty area three minutes later.
McLaughlin grabbed his second just after the hour mark to wrap up the victory, before Jonathan Smith fired home after some good work from Nacho Novo to give Glentoran a consolation goal.
Defeat for Glentoran means that the East Belfast side slip into the bottom half of the table, while Coleraine's first victory of the season carries Oran Kearney's side up to eighth.
Ballymena responded from last weekend's defeat by Ards in perfect style, putting four goals past Carrick Rangers.
Allan Jenkins opened the scoring, slamming home from a corner after just three minutes.
Cathair Friel doubled Ballymena's advantage on 22 minutes by heading in from a Tony Kane free-kick.
Carrick Rangers were provided with a glimmer of hope when Alan Blayney picked up a back-pass and Gareth McKeown's free-kick rebounded off the United wall for Nathan Hanley to make no mistake from the rebound.
Jonny McMurray scored Ballymena's goal of the evening, letting fire with a fine 30-yard strike which flew over Simon Steele in the Carrick goal.  Ballymena completed the rout when former Carrick player Conor McCloskey slotted home just after the hour mark.
The developer of a new retail area in Dorchester town centre has pulled out of the scheme.
[NEXT_CONCEPT]
Glenavon beat Ards 1-0 to join the newly promoted Co Down side at the Premiership summit.