HM Inspectorate of Prisons (HMIP) said assaults at HMP Leeds had increased significantly since 2013.
On average there are now 32 assaults on inmates and staff each month, of which about 10% result in serious injury.
The National Offender Management Service said it was taking "concerted action" to combat legal highs.
HMIP said new psychoactive substances - also known as legal highs - were a "major factor" in the increase in violence.
Inspectors also found the use of force and segregation at the prison, known locally as Armley jail, was "high" and the majority of cells were "poorly equipped".
Levels of crowding were also found to be "very high", with 1,149 inmates held at the time of the inspection in December against an operational capacity of 1,219.
However, inspectors found support for men with drug and alcohol problems was "generally good", prisoners spent a "reasonable time" out of their cells and the staff culture was "basically positive and decent".
It also said that given the age of the prison - HMP Leeds was built in 1847 - the accommodation was "reasonably well maintained and the environment was clean".
Martin Lomas, deputy chief inspector of prisons, described the inspection results as "disappointing" and said fundamental issues around safety needed to be addressed "urgently".
However, he said the new governor had made a "good start in getting to grips with these challenges" and had a "good understanding of the issues faced".
Michael Spurr, chief executive of the National Offender Management Service, said HMP Leeds, like other publicly-funded prisons, had to cope with a "huge increase" in the supply of legal highs against a backdrop of cost-cutting.
He said improving safety was the "number one priority" and the governor and staff would "receive the support they need to make the improvements required".
Juliet Lyon, director of the Prison Reform Trust, said the prison was failing to meet "people's most basic need - safety".
"How can we expect people to leave custody to be less likely to offend, if all they have experienced in prison is fear and violence?"

This breaking news story is being updated and more details will be published shortly. Please refresh the page for the fullest version.
If you want to receive Breaking News alerts via email, or on a smartphone or tablet via the BBC News App then details on how to do so are available on this help page. You can also follow @BBCBreaking on Twitter to get the latest alerts.
Legal highs are a "major factor" behind a sharp rise in rates of violence at a prison to almost double that of similar jails, a report has found.
[NEXT_CONCEPT]
Discussions continue in court over where terminally ill baby Charlie Gard should spend his final days.