Younger people are leading a trend towards buying products without dairy and gluten in particular.
UK supermarkets are increasingly stocking aisles with wide ranges of the products, from biscuits to bolognaise.
"Consumers are associating 'free from' with a natural form of health in general," said Fraser McKevitt of the consumer research firm.
He said that while only about 5% of the population needed to avoid certain food groups for medical reasons, 54% of households joined the trend in the first three months of this year by buying at least one product from the category.
In recent years, a fashion for "clean eating", paleo, vegan and other diets popularised online has boosted the market for specially manufactured products and encouraged supermarkets as well as specialist stores to stock more of them.
Compared with last year's survey, 3.3 million more people had purchased a "free from" product, said Mr McKevitt.
Households containing people aged under 45 were 20% more likely to buy from these ranges. Sales of "free from" products grew 36% compared with a year earlier, he said.
Kantar monitors shoppers' habits using a sample of 30,000 households.
"The majority of products are bought by people who do not identify as having an intolerance," said Mr McKevitt, head of retail and consumer insight at Kantar.
He said shoppers used the label as a short-cut to identify products they believed would be healthier.
Ursula Philpott, a dietician at Leeds Beckett University, said that for the section of the population that did suffer from allergies and intolerances, the increased availability of these products was very helpful, but most of people did not need to avoid any specific food groups.
"A lot of people will be buying these because they've been sold the idea that gluten or dairy are in some way bad for you and that that's science-based, which for most people just isn't the case," she said.
"If we look at studies over the long term, we know the types of diets that lead to us having longer lives or having less disease states are things like Mediterranean diets, which don't include cutting out lots of things."
Kantar Worldpanel's survey also indicated that sales of "free from" products were higher amongst a more affluent demographic, reflected in higher sales of the products at Waitrose and Sainsbury's, compared with those grocers' market share.
But discounters Aldi and Lidl were also selling significant amounts of these products, said Mr McKevitt.
"Meanwhile, inflation shows no signs of abating. The price of everyday goods is up by 2.3% compared to this time last year, and rising prices cost the average household an additional Â£21.31 during the past 12 weeks," he added.
"We expect inflation to continue to accelerate, and as a result, we're likely to see consumers looking for cheaper alternatives."
The latest figures from Kantar Worldpanel indicate that prices for butter, fish, tea and skincare have risen in the 12 weeks to 26 March, while categories including crisps, bacon, chocolate and fresh poultry have seen prices fall.
Discount retailers Aldi and Lidl have continued to increase sales by about 15%, whereas Tesco, Sainsbury's and Asda saw their sales decline slightly. Iceland has also seen a boost to sales.
Aldi and Lidl now account for 11.7% of the UK grocery market.
More than half of us bought a "free from" product during the last three months, according to Kantar Worldpanel.