The failure of BHS and tough competition in the clothing market contributed to the poor performance.
A report filed with Companies House by Taveta investments showed pre-tax profits for the 12 months to August 27 2016 fell to £36.8m, down from £172.2m the previous year.
Total sales fell 2.5% to £2.02bn.
Taveta is the holding company of Arcadia which alongside its biggest brand, Topshop, owns Miss Selfridge, Burton, Evans, Wallis and Dorothy Perkins.
Taveta's statement reveals its profits were dented by £129.2m in one-off charges.
These exceptional costs included £26.4m related to the collapse into administration of BHS in April of last year, and the "subsequent regulatory investigations", it said.
Taveta also made a £21.8m provision for "onerous" leases on loss-making stores last year.
It said its financial performance was "below prior year levels".
"The retail industry continues to experience a period of major change as customers become ever more selective and value conscious and advances in technology open up more diverse, fast-changing and complex sales channels.
"Clothing has also become a less important part of the household budget," it added, echoing recent statements made by other High Street fashion chains.
Meanwhile, Taveta said Brexit had caused economic uncertainty and the fall in sterling had dented its results.
"The group is looking at initiatives to improve margin to offset the ongoing impact of weaker sterling."
Market conditions remained "challenging and very competitive", with new players entering the market particularly online, it continued.
And while UK unemployment was low and consumer credit availability continued to rise "the slow growth in average earnings impacts the spending power of customers".
The last BHS store closed its doors last year when no buyer could be found for the chain.
There was an estimated £571m hole in the BHS pension covering all future payouts. The potential of taking on such a burden was seen as one of the reasons that BHS failed to find backers or buyers for the business at a whole.
In February of this year Sir Philip agreed a £363m cash settlement with the Pensions Regulator to plug the gap in the BHS pension scheme.
The deal means workers will get the same starting pension that they were originally promised.

The 22-year-old won his first medal in the 10m platform event, finishing with a score of 543.80.
China's Olympic champion Aisen Chen took gold with 580.60 while silver went to compatriot Hao Yang (574.80).
Daley then teamed up with 20-year-old Grace Reid to finish third in the mixed 3m synchro behind pairs from China and Canada.
Their best dive came in the third round when they scored 69.30 points for their inward two-and-a-half somersaults piked.
Profits at Sir Philip Green's retail empire, which includes High Street fashion chain Topshop, plummeted by 79% last year.
[NEXT_CONCEPT]
Britain's Tom Daley won two bronzes on the final day of the Diving World Cup event in Kazan, Russia.