Agencies had insufficient knowledge and understanding of minority and faith groups, a serious case review said.
Abdullah and Jaffar Deghayes were in a child protection plan before 2010, the report by a senior social worker said.
But the review found their radicalisation was a "total shock and surprise" to authorities in Brighton.
Abdullah, 18, and Jaffar, 17, were both killed having followed their older brother, Amer, to Syria to fight for an Al-Qaeda-affiliated jihadist group.
Both boys had suffered bullying and racism, and had reported physical abuse by their father.
Who are Britain's jihadists?
Live: More on this story and other news from Sussex
The report by Edi Carmi said the brothers were taken out of the child protection plan in 2010 because there was professional "helplessness" among social workers about what else to do.
The report said this was expressed by some workers as having "no tools in the toolbox".
It concluded that professionals often lack "effective ways to intervene in families who have suffered long-standing trauma".
In the years that followed the end of the protection plan, Abdullah, Jaffar and Amer showed signs of radicalisation.
In early 2013, a school reported concerns about some young people including Jaffar.
One of the brothers further came to the attention of social workers over an "emotional" comment he made about "Americans" after he returned from a trip to his family's home country of Libya.
This led to a referral to the "Channel panel" - a de-radicalisation process - but it was decided he was "not at risk of being drawn into terror-related activities".
Graham Bartlett, independent chair of the Brighton and Hove Local Safeguarding Children Board, said: "The system as a whole let these young boys down. It's a wake up call.
"This case has had a major impact on our understanding of the risks posed to children of exploitation through radicalisation."
The report identified 13 key findings, including that professional responsibilities springing from the government's counter-terrorism strategy were not fully understood by all staff.
It also said professionals had no effective ways to intervene in families who have suffered long-standing trauma, and local statutory agencies had "insufficient knowledge about, and understanding of, local minority ethnic and faith community groups and how best to work together to safeguard children".
The Safeguarding Children Board said it fully accepted the report's findings.
Opportunities were missed to spot the radicalisation of two teenage British Muslim brothers who died fighting in Syria in 2014, a report has found.