To avoid the police, the spot found for the fight was on a piece of meadow ground in a hollow where the contestants and supporters could scarcely be seen. Broome had lost as much as 40 pounds for the contest, but did not appear as muscular as he had in earlier contests. His determination was substantial, but his stamina was described as inadequate against the vigor and strength of his opponent. By the 48th round, the left side of Broome's face was quite swollen, and his left eye nearly swollen shut. Broome fell in the 49th from a blow to his left ear. In the brutal affair, Broome was down again in the 50th. In the 51st and final round, Broome was down from a right to the chest, and though he tried to rise, his second threw in the sponge. Though the betting favorite early on, Broome was described as "completely disabled" and presented a "shocking spectacle" by the end of the fight. He attempted to retire immediately after the disabling loss. After finishing his fighting career he kept a pub in Portsmouth and later kept the "Crown and Cushion" in Little Russell Street, London. In later years, he was often found wagering money at race meetings which may have contributed to his poverty in later life. Near the end of his life, he suffered for months from lung disease and "dropsy", now known as edema. After being released from hospital care, he died in poverty due to his medical expenses, on the evening of 2 November 1865 at his residence on 30 Frith Street, in London's Soho. He was survived by his wife, who cared for him during his illness, and two children. He was buried in Norwood Cemetery in the same tomb as his brother, Johnny Broome, requiring patrons to pay the sum of £4 for the burial.