Islamic schools that promote "isolationism" and extremism should be closed, Communities Secretary Ruth Kelly has said.
Ruth Kelly said she had regularly met with Muslim groups
But people should not be threatened by symbols of faiths such as Islam, Mrs Kelly told the BBC.
She added the government was opposed to any form of law contrary to British civil law, such as Sharia law.
Mrs Kelly also expressed surprise at a decision to caution a football player who crossed himself during a match.
She said the government had to "stamp out" Muslim schools which were trying to change British society to fit Islamic values.
"They should be shut down," she said. "Different institutions are open to abuse and where we find abuse we have got to stamp it out and prevent that happening."
Muslim communities needed help to combat radicalism, Ms Kelly said
But she said Muslims were entitled to the same rights as Anglicans, Catholics, Hindus and Jewish groups which all had faith schools.
She said that Britons should not be scared of Muslims observing their faith.
"When I see a Muslim woman wearing a headscarf or a hijab, I don't feel threatened, I celebrate it."
Ms Kelly's comments followed the launch last week of the government's Commission on Integration and Cohesion, in which she urged a "new and honest" debate on diversity.
Ms Kelly said she did not believe the vast majority of British Muslims supported attacks on this country.
"Most Muslims would call those terrorists who would undermine the fabric of this society as not true Muslims but revolutionaries who are cowering under the cloak of Islam."
The commission, originally mooted last July in the wake of the London terror attacks, came amid growing fears of alienation, especially among young Muslims.
Ms Kelly said the government had to work with "these law-abiding Muslims people in this country who try to combat that".
She said she had met with many members of the Muslim community, including women's groups and young people as well as more traditional Islamic community leaders to deal with the threat of radicalised British Muslims.
"The bottom line is, if they are glorifying terrorism, if they are criminals, if they are breaking the law in this country then they should be arrested and dealt with appropriately.
"We've got to say that clearly and not be afraid to say it, and work with the community and the Muslim community as well."
Ms Kelly said the government would not allow any other form of law - such as Islamic Sharia law - to be legal for family use.
"We are not going down that route. We don't think that's compatible with Britain being a tolerant, diverse society that welcomes people of different faiths."
Ms Kelly also spoke of her surprise at Scottish prosecutors' decision to caution Celtic Artur Boruc over making the Catholic sign of the Cross at a game against Rangers in February.
"This, traditionally, has been a country which has valued religious diversity - and cultural and racial diversity as well - and where there has been freedom of expression both to express religious symbols but also other cultural symbols as well," she said.
In addition to crossing himself, the player was alleged to have made gestures to the crowd at the start of the second half of the game on 12 February.