Jack Straw, the ex-foreign secretary, has angered Muslim groups by suggesting women who wear veils over their face can make community relations harder.
Mr Straw says covering faces makes good community relations difficult
The Blackburn MP says the veil is a "visible statement of separation and of difference" and he asks women visiting his surgery to consider removing it.
The Islamic Human Rights Commission said the Commons leader's request was selective discrimination.
But the Muslim Council of Britain said it understood Mr Straw's discomfort.
Muslims make up about a quarter of the population of Blackburn, and Mr Straw said he had carefully considered his remarks.
He wrote in the Lancashire Evening Telegraph that he feared "wearing the full veil was bound to make better, positive relations between the two communities more difficult".
Asking women to consider showing the mouths and noses could lead to true "face-to-face" conversations with constituents, enabling him to "see what the other person means, and not just hear what they say".
He said he made sure he had a female colleague in the room when asking someone to show their mouth and nose - and his constituents had so far always agreed to do so.
Later Mr Straw, who has defended the right for women to wear headscarves, asked BBC Radio Lancashire: "Would those people who do wear the veil think about the implications for community relations?"
The remarks attracted an angry response from some organisations representing Muslims.
It was "astonishing" that Mr Straw chose to "selectively discriminate on the basis of religion", said Massoud Shadjareh, chairman of the Islamic Human Rights Commission.
Halima Hussain, from civil liberties group the Muslim Public Affairs Committee, asked BBC News 24: "Who is Jack Straw to comment on negative symbols within a religion that is not his own?"
Rajnaara Akhtar, who chairs the organisation Protect-Hijab, suggested the "appalling" comments showed "a deep lack of understanding".
Mr Straw was putting women "into a very awkward position by compromising the faith they believe in and that is ill-placed", Council of Lancashire Mosques chairman Hamid Kureshi told BBC Radio Five Live.
And a political rival - Liberal Democrat constitutional affairs spokesman Simon Hughes - questioned whether it was Mr Straw's place to question the way that members of the public dressed.
"I don't think it's the job for somebody who represents the whole community to say to somebody who comes through the door, 'Do you mind if you dress differently in order to talk to me?'," Mr Hughes said.
Oliver Letwin, the Conservatives' policy chief, told the BBC's Question Time programme that if women wanted to wear a veil they should do so. He described it as "dangerous" to suggest they should not be allowed to.
Labour chairman Hazel Blears called for a debate among Muslim women on the issue, but said it was a constituency matter rather than one for the government.
"I do not think it's right for government to lay down laws about what people should wear and what they shouldn't," she said.
Jack Straw said he had carefully considered his remarks
"Jack was dealing with his constituency issues. He has a very large Muslim population and he is very close to that community."
Labour MP for Dewsbury Shahid Malik said Mr Straw was entitled to his views and called for increased understanding between Muslims and non-Muslims.
Dr Daud Abdullah of the Muslim Council of Britain said individual Muslim women could choose to remove part of their veil.
"Even within the Muslim community, the scholars have different views on this.
"Our view is that if it is going to cause discomfort and that can be avoided then it can be done."
Dr Abdullah added, however, that covering hair remained "obligatory" for Muslim women.
Mr Straw was home secretary from 1997 to 2001, and then foreign secretary until 2005, a period which included the build-up to, and invasion of, Iraq.