Barry Imray, 35, of no fixed address, denies killing Lee Irving, who was found dead near a footpath in Newcastle last year.
Another man, James Wheatley, 29, from Studdon Walk, also denies murder.
Mr Wheatley's mother, Julie Mills, 51, and his girlfriend, Nicole Lawrence, 22, deny "causing or allowing" Mr Irving's death.
Mr Imray told Newcastle Crown Court he had not harmed 24-year-old Mr Irving, but panicked when he realised he was not moving, and alerted the others.
He denied that he had moved Mr Irving's body as a result of being responsible for his death.
He said he had taken the body away from the house all five shared in Kenton Bar because he had been told to.
When asked by Mr Wheatley's defence barrister, Francis Fitzgibbon QC, why he had not refused, he said: "You can't say no to James [Wheatley]."
The jury heard that Mr Imray accepted he had lied when he told police he had merely come across Mr Irving's body by the footpath. He lied because he was frightened, he said.
It is alleged that Mr Irving, who had learning difficulties, had been sedated by the defendants, rather than taken for medical treatment.
The court has heard he was found about half a mile from the house with 27 rib fractures and a broken nose and jaw.
The trial continues.
A man accused of murder has told a court he was told to move the body and was too scared to refuse.