An inspection of Swinfen Hall prison near Lichfield, Staffordshire found almost all areas have deteriorated and the prison is "not safe enough".
The report also found inmates at the prison and young offenders institution were not having daily showers.
A spokesperson for the prison said the problems reflected "operational pressures".
The Chief Inspector of Prisons, Peter Clarke, said: "Basic standards to improve".
The report by HM Inspectorate of Prisons found:
The Chief Executive of the Howard League for Penal Reform charity, Frances Crook, said: "For the second time in as many weeks, we read of a prison where men are so frightened for their safety because of rising violence that they are refusing to come out of their cells.
"Many prisons are overcrowded, but Swinfen Hall is not, which makes this report particularly concerning."
Michael Spurr, Chief Executive of the National Offender Management Service, said more prison officers will be recruited to Swinfen Hall and an action plan is in place.
Inspectors noted the prison was working hard to address offending behaviour and work to resettle prisoners back into the community was "reasonably good".
In August 2015, the prison was put on lockdown. In December of the same year prison staff were taken to hospital after being attacked by inmates.
The unannounced inspection of the prison, which holds 600 male prisoners aged between 18 and 25, was carried out in October and November 2016.
Inmates are living in cells without glass in the windows in conditions described as "squalid".