The club had hoped to build a 21,000-seater venue at Stoke Gifford on the outskirts of the city.
It said it was "disappointed" that discussions with UWE had "come to a close as the parties were unable to agree acceptable terms".
The university insisted it had "not given up on this hugely important project".
In 2011, the football club reached a deal with UWE and Sainsbury's to build a new stadium on university land.
Further plans included the supermarket building a store and redeveloping the club's Memorial Stadium in Horfield.
However, the chain pulled out following delays and petitions by campaigners for and against the plans. A High Court battle ensued which Rovers lost in 2015. The club was bought the following year by the Al-Qadi family.
In a statement, Rovers said it would "now have to go back to the drawing board" and explore options for a new stadium, redevelopment of its current ground and improvements to its Colony training ground.
Club manager Darrell Clarke said the news was "very disappointing for everybody".
Rovers chairman Steve Hamer, added: "I can't see this position being retrieved.
"I just think this whole deal has been jinxed, it just never seems to have been running in the right direction."
A UWE spokeswoman said the university "remained committed" to a new stadium, which would bring "great benefits" to the institution, community and regional economy.
She added: "Although we have been disappointed by the slow pace of negotiations, we are still willing and open to continue negotiations.
"This has been an initiative the university has been working on for over eight years and we still believe is deliverable.
"Planning consent does not expire until January 2018 and the door remains open for a deal to be struck, and the university will continue to work on this project."
With so much positivity at Bristol Rovers going into the 2017-18 season, with a number of signings and manager Darrell Clarke signing a five-year contract, this news will come as a blow - especially given the time and money spent on the project.
Supporters have waited patiently for progress since the plans to build a new stadium at UWE were first released in 2011. Indeed, you could go back even further, to over three decades ago when they left Eastville.
Now fans will want to know what happens next and they'll want reassurances that the Al-Qadi family remain committed to the football club.

An exhibition in London explores the global refugee crisis through the eyes of eight photographers reporting from 13 countries on behalf of Oxfam.
The pictures aim to capture something of the personal stories behind the migration facts and figures.
Photography director Rebecca McClelland said: "My aim was to be as sensitive as possible to the subject, whilst stretching the limited language of crisis photography."
The Stand as One exhibition can be seen at 4 Holywell Lane, London from 8-15 September 2016.
Bristol Rovers has abandoned plans to build a new stadium at the University of West of England (UWE).
[NEXT_CONCEPT]
All photographs courtesy Oxfam