Poorer people regard smoking as one of the few pleasures they have access to, Health Secretary John Reid has said.
Dr Reid's comments are in contrast to those of the prime minister
His comments come days after Tony Blair said ministers were considering banning smoking in public places.
Ex-smoker Dr Reid said: "All I say is be careful, please be careful that we don't patronise people.
"As my mother would put it, people from those lower socio-economic categories have very few pleasures in life and one of them they regard as smoking."
He added at a Labour Big Conversation event in London: "I note the forcible representations on banning smoking, particularly banning smoking to overcome the difficulties that the lower working class get out of the ailments of smoking.
"I just worry slightly about the unanimity of the medical and professional activists in taking that view."
Dr Reid said his fears about the issue were not allayed when he heard one contributor to the event saying "we should start by banning 10 cigarettes so those who can only afford 10 can't have a fag".
He asked: "But if you can buy them in a 100 presumably it's alright?"
Later in a statement he added: "We want everyone to live a healthy lifestyle but not everyone lives in the same circumstances.
"If we wish to change people's habits we will often have to help change the circumstances in which they live."
Dr Reid's comments were seized upon by the Tories who branded them "regrettable".
Tory health spokesman Andrew Lansley said: "It is impossible to see how the government can promote a consistent public health strategy when with one hand it is funding the British Heart Foundation's ad campaign against smoking and with the other John Reid makes remarks likes these."
He added: "To suggest that for a poor mum with three kids to be smoking is anything other than damaging, coming from the health secretary, is regrettable."
Lib Dem health spokesman Paul Burstow said: "John Reid's message to those in deprived areas is not 'let them eat cake' but 'let them smoke fags'.
"This is yet more evidence that the Health Secretary has no clue when it comes to public health. His statement is patronising, damaging and based on weak assumptions."
The prime minister's spokesman later insisted the health secretary had been contributing to the debate on the issue being encouraged by the government.
"We want everybody to pursue a healthy lifestyle. There are warnings and everybody knows the dangers of smoking.
"At the moment we are having a discussion and people are expressing different views. The advice is there, the health secretary has made his comments and the consultation will continue."
That consultation should be completed by the summer when the government would decide what to do.
On Friday the prime minister said the government was considering introducing a ban on smoking in public places and will come to a view in the "next few months".
He stressed it was "a difficult balance" protecting the public's health on the one hand and not being overly interfering on the other.
Dr Reid went further indicating that he was not in favour of instructing ordering adults about how to make their choices.
The health secretary's comments came under fire from health and anti-smoking organisations.
The British Medical Association expressed surprise at what Mr Reid had said
"Quite apart from the individual damage to smokers, there's passive smoking to consider," a spokesman said.
And a spokesman for the anti-smoking group Ash said: "It's incredibly patronising to talk about smoking in this way - the argument is that we should have smoke-free work environments."
He added: "Smoking kills a disproportionate number of people from social classes D and E.
"And it's the biggest single contributor to health inequality - and differences in life expectancy - between social classes."
But the health secretary's remarks were welcomed by smokers' lobby Forest which said smoking gave a lot of pleasure to a lot of people.
On Friday Mr Blair told BBC Breakfast: "You have got to have some balanced decision making in this, and it's a difficult balance."
The government has already announced it is assessing the public's feelings about a ban as part of a major consultation on health that will form part of the Public Health White Paper, to be published later this year.
But the prime minister hinted the decision might ultimately be left to local authorities.
"In the end, though, you have also got to have some local decision-making in
this," he said.
Smoking is the biggest cause of preventable death in the UK, responsible for 120,000 premature deaths a year.