Prison officers watched daytime television rather than deal with prisoners, inspectors have said.
Whitemoor Prison is home to Category A and B prisoners
Chief Inspector of Prisons, Anne Owers, said her team was surprised at officers relaxing during their five-day visit to Whitemoor Prison at March in Cambs.
In one case Whitemoor staff finished viewing a programme before turning their attention to a waiting prisoner.
Whitemoor, which houses dangerous offenders, was described as "controlled and orderly" by the Prison Service.
But Ms Owers said: "In one case a prisoner was ignored after complying with a request to attend the office, while staff watched television as he waited.
"Towards the end of the inspection we noticed that televisions were switched off, at least while we were on the wings."
Some staff treated inmates and talked about them in "dismissive ways", the report said.
"It was routine to see conversations taking place only through bars, and prisoners being ignored when seeking attention."
On the positive side, specialist units - such as the dangerous and severe personality disorders wing - were said to be performing well.
However, black and Asian prisoners had reported "subtle discrimination" which the report suggested was due to a lack of understanding by the "almost entirely white staff" at the 500-inmate jail.
Healthcare was also poor, with chronic staff shortages and insufficient services for mental care cases.
Management of the pharmacy was described as "lamentable and bordering on the dangerous".
Ms Owers said: "The enthusiasm and active engagement of staff in the specialist units had not sufficiently fed back into the prison as a whole.
"Management of key areas such as healthcare, activities and sentence planning required improvement.
"There were signs that these weaknesses were being addressed, and there is much good practice to build on."
Prison Service director general Phil Wheatley said: "Whitemoor is a Category A prison with a diverse population including many of the most criminal and dangerous prisoners in our custody.
"It consistently delivers high security and is a controlled and orderly prison."
A new pharmacy contract was now in place, he added, and the healthcare centre was due to be fully staffed by the autumn.