An HM Inspectorate of Prisons report into the Verne IRC in Portland also criticised the amount of legal representation available to detainees.
The report praised treatment of detainees by staff and the use of body cameras by managers.
The Home Office said it took detainees' welfare "extremely seriously".
HMP Verne was reclassified as an immigration removal centre in September 2014.
Almost 600 men awaiting deportation were held at the centre, with a majority having previously been in prison following criminal convictions.
During what was the first announced inspection since the change in status, inspectors found "levels of violence were too high and some of the violence was serious".
They reported a "concerted indiscipline" in the week before their visit in March.
They also pointed to higher than usual availability of new psychoactive substances and illicit alcohol.
The inspectors found 30 minutes of free legal advice was available and many detainees "struggled to obtain representation to fight their cases".
Their report also highlighted "excessive stays", including almost 40 men being held for more than a year, and one held for five years, which inspectors described as "one of the worse cases of prolonged detention we have seen".
It also said the former Victorian fort complex remained "too prison-like in character", with too much inner fencing and razor wire and restrictions on movement.
Chief Inspector of Prisons Nick Hardwick said: "Overall, The Verne was operating satisfactorily. However, despite considerable efforts to prepare the institution for its new role, the environment and staff culture reflected an institution that had not yet come to terms with its new function as an IRC."
A Home Office spokesperson said: "We are pleased the inspector has reported that the majority of detainees in The Verne felt safe and were very positive about their treatment by staff.
"We have considered the inspector's recommendations carefully and are committed to working with our contractors to implement our service improvement plan for the centre."
Former prisons ombudsman Stephen Shaw is to complete an independent review of detainees' welfare expected to be completed in the autumn.

The Syrian leader also said the army faced a shortage of soldiers.
A day earlier, he declared an amnesty for draft-dodgers and deserters.
The conflict is thought to have left more than 230,000 dead and displaced millions. Vast areas are no longer under government control.
Syria's conscript army was once 300,000 strong, but has been roughly halved by deaths, defections, and a rise in draft-dodging, AFP news agency said.
The Syrian leader said the army did not have the manpower to defend the entire country, especially as rebel groups were receiving increased support from outside - a reference to Saudi Arabia, Turkey and Qatar.
"Sometimes, in some circumstances, we are forced to give up areas to move those forces to the areas that we want to hold onto," Mr Assad said in a televised speech to dignitaries in Damascus.
"We must define the important regions that the armed forces hold onto so it doesn't allow the collapse of the rest of the areas."
The BBC's Jim Muir - in neighbouring Lebanon - says it has been evident for some months that Syrian government forces have been focusing on defending certain key areas while not doing everything they might to defend outlying regions where the army is really overstretched.
This year, the Syrian military has lost the north-western provincial capital Idlib, parts of the south, and Palmyra in the north-east, which was taken over by militants of Islamic State.
Bashar Assad said he was sure the army could defend the core areas where it was consolidating - meaning Damascus, the cities of Homs and Hama and the coast.
But other big cities such as Aleppo in the north and Deraa in the south may come under question, our correspondent says.
However, Mr Assad pledged to fight on and ruled out the prospect of any negotiated settlement at the moment.
"The word defeat does not exist in the Syrian army's dictionary," he said, adding that  "collapse" was not on the cards.
"We will resist and we will win."
The Syrian army, which is fighting rebels and jihadist groups, began a recruitment drive at the start of July to try to tackle its manpower shortage.
At least 70,000 men have avoided military service, according to the Syrian Observatory for Human Rights.
More than 80,000 soldiers and pro-government militiamen have been killed since the start of the conflict in March 2011.
High levels of violence were recorded at an immigration removal centre (IRC) in a former Dorset jail, the prison watchdog has found.
[NEXT_CONCEPT]
The Syrian army has been forced to give up some areas in order to retain others in the war against rebels, now in its fifth year, President Bashar al-Assad has acknowledged.