The Rail Accident Investigation Board (RAIB) report blamed the near miss at Hest Bank, Lancashire, last September on the lack of warning from a lookout.
Four of the group working at the bridge had to cling to its parapet to keep clear of the train, said the report.
Network Rail said: "Our railway is the safest in Europe."
"We are always looking at ways to improve safety for the benefit of those who work on the railway and those who travel on it," a spokesman added.
The RAIB report said the party, who were working on repairing a sleeper, should have received a warning from a lookout.
Instead the controller of site safety had to shout an urgent warning after spotting the train, travelling from Edinburgh to Manchester Airport, when it was just three seconds away.
"Staff took rapid evasive action, but some members of the workgroup were on the bridge and were unable to reach the designated position of safety, instead pressing themselves against the bridge parapet," said the report.
"The last member of the workgroup was not clear of the line until about one second before the train passed."
The report said the lookout might have pushed the wrong switch on the electronic system to warn colleagues, might have forgotten to send a warning or their "vigilance had degraded" because it was near the end of the recommended two-hour shift for a lookout.

Algerian TV broadcast footage of Mr Bouteflika's meeting with the prime minister and head of the armed forces a day earlier in the French capital.
He was seen in a black dressing gown, talking, drinking from a coffee cup, and eating a pastry.
Mr Bouteflika's absence has fuelled speculation about his condition.
In the footage broadcast on Wednesday, the 76-year-old leader was seen during a meeting at the National Residence of Invalids, a complex built for war veterans that includes a hospital.
Mr Bouteflika was taken there after being treated in Val de Grace military hospital.
He looked frail, talking and smiling occasionally and moving his right hand.
Algerian state media said he talked with Prime Minister Abdelmalek Sellal and Chief of Staff Ahmed Gaid Salah for two hours, covering the draft budget and other government business that needed to be approved by the cabinet.
Mr Bouteflika gave "detailed instructions" over the supply of goods ahead of the holy month of Ramadan, which begins in July, APS news agency reported.
It also issued a health bulletin saying that the president had suffered a transient ischaemic attack on 27 April that had "not affected his vital functions".
It said doctors had "recommended that he observe a period of convalescence and functional rehabilitation to consolidate his recovery".
Mr Bouteflika has been in power since 1999. His latest health problems are widely seen to have ended his chances of running for another term in presidential elections next year.
Nine railway workers were within seconds of being hit by a train travelling at 98mph, a safety investigation has found.
[NEXT_CONCEPT]
Images of Algerian President Abdelaziz Bouteflika have been shown for the first time since he went to Paris to be treated for a mini-stroke in April.