Shadow Europe minister Pat Glass made the comments after an interview with BBC Radio Derby in Sawley, Derbyshire.
She said: "The very first person I come to is a horrible racist. I'm never coming back to wherever this is."
She later said the comments were "inappropriate" but UKIP said it showed "contempt" for people's concerns.
Updates on this story and more from Derbyshire
The man Ms Glass is believed to have been referring to said he had spoken to her about to a Polish family in the area who he believed were living on benefits, describing them as "spongers", but denied being racist.
The North West Durham MP said: "The comments I made were inappropriate and I regret them.
"Concerns about immigration are entirely valid and it's important that politicians engage with them.
"I apologise to the people living in Sawley for any offence I have caused."
But those campaigning for the UK to leave the EU criticised her remarks.
Conservative MP Peter Bone, one of the founders of Grassroots Out, said: "It is disappointing that a member of the Remain campaign called a voter a 'horrible racist' for raising perfectly valid concerns about the impact uncontrolled migration is having on the UK."
And Steven Woolfe MEP, UKIP's migration spokesman, said her comments showed "the contempt Labour has for ordinary workers' concerns".
"It is in the DNA of the Labour Party. We have had Gordon Brown calling a Labour supporter a bigot, Emily Thornberry ridiculing a white van man displaying an English flag - and now this."
Brendan Chilton, the general secretary of the Labour group campaigning for Brexit - Labour Leave - criticised Ms Glass's comments as "shocking". "A significant number of Labour voters want to leave the EU, and have justified concerns with immigration.
"Pat Glass's comments do not reflect the views of a large number of Labour voters."
Conservative MP for Erewash Maggie Throup had urged Ms Glass "to make a full apology to my constituent at the earliest opportunity" and to return to the area "to see for herself just how great a place it is to live, work and raise a family".
In April 2010, then PM Gordon Brown apologised after being recorded describing a 65-year-old voter as a "bigoted woman".
Labour MP Ms Thornberry quit the shadow cabinet in 2014 after a photo she tweeted of a house with three England flags and a white van parked outside, during the Rochester by-election, was criticised as "disrespectful".
Shadow home secretary Yvette Cooper, who is campaigning for the UK to remain in the EU, told BBC One's Question Time: "Pat has apologised and of course it's not racist to want to leave the European Union, it's not racist to be worried about immigration - there are concerns particularly about dodgy employers who exploit low skilled migration to undercut wages and jobs and I think something should be done about that."
But she accused Leave campaigners of offering "false promises" that if the UK left the EU there would be a "substantial" change on migration policy.
A Labour MP campaigning for the UK to remain in the EU has apologised after being recorded calling a voter a "horrible racist".