Too many politicians are "piling in" to the debate on Muslim women who wear full-face veils, the Tory leader says.
Mrs Azmi criticised ministers who had intervened in her case
David Cameron told ITV1's Frost Tonight he was concerned British Muslims were left feeling "slightly targeted".
But Mr Cameron said he sympathised with the school that suspended a teaching assistant who wore her veil in class.
On Thursday, Aishah Azmi lost her religious discrimination and harassment claim but Kirklees Council was ordered to pay £1,100 for victimising her.
Social Exclusion Minister Hilary Armstrong added her voice to the debate by telling BBC One's Question Time that wearing the full veil could present difficulties.
She said: "Those who decide to wear the veil just make it that much more difficult for their neighbours, anybody that they're talking to, to really feel that they are sharing values and so on with them," she said.
"I think it is very difficult to actually wear a veil and participate in everything in our society."
Following the tribunal ruling, Mrs Azmi, who was suspended from Headfield Church of England Junior School, in Dewsbury, West Yorkshire, had attacked ministers for commenting on the case.
The head of the Muslim Council of Britain, Muhammad Abdul Bari, has also complained that a "drip feed" of ministerial statements on the veils issue has "stigmatised" the entire Muslim community.
A number of politicians have aired their views since Commons leader Jack Straw said earlier this month that he had asked women to remove full-face veils in his Blackburn constituency advice surgeries.
He argued they could make community relations more difficult.
Mr Cameron told Frost Tonight: "I think there is a danger of politicians piling in to have their tenpence-worth and really they have to ask themselves whether this is having an overall good effect or not."
He added that he believed Mr Straw he had raised the issue in a "calm, reasonable, moderate" way.
However, the Tory leader said: "I am not sure that if someone came to my surgery wearing a veil I would ask them to take it off.
"I don't think that's what I'm like. But I thought [Jack Straw] put the points reasonably."
Mr Cameron said that in cases like Mrs Azmi's, it should be up to the school and local authority to make their own judgement.
"I am not sure we can have some national veils policy," he said.
But he added: "I have great sympathy with the school because it seems to me there isn't a teaching in Islam which says you have to wear the veil in front of children and, in terms of teaching, communication is vitally important."
Jim Dodds, a cabinet member for children's services at Kirklees Council, which is the local education authority in that area, told the BBC he believed the tribunal ruling was right.
He said the council had taken its stance because the children were "our paramount responsibility".
"For a teacher or support staff that was teaching children a language, the children must be able to see the lips and the words being formed.
"It was... a common-sense approach and it was about education and what was best for the children, nothing more and nothing less."