But the Premier League club say they want Allardyce to stay, insisting he is "very much key to our plans".
Sunderland claim speculation about Allardyce has been "extremely damaging" and have urged the FA to "bring about a swift resolution to the matter".
England are searching for a new boss after Roy Hodgson quit on 28 June.
He resigned after the national team were knocked out of Euro 2016 at the last-16 stage by Iceland.
Photographs that appeared to show Allardyce, 61, at the home of FA vice-chairman David Gill were published in the media in the past 24 hours.
Allardyce had been on tour with Sunderland in Austria until he returned home at the start of the week, reportedly on transfer business.
Sunderland said they agreed to let Allardyce speak with the FA as a "potential candidate" after he requested permission, but are upset discussions did not remain confidential.
"After what was an extremely challenging season, we are keen to see a period of stability, both on and off the field, and we want him to remain as manager of our football club," read a club statement issued on Wednesday.
Kevin Davies, who played under Allardyce at Bolton, thinks his former boss would be a good choice for England.
"Where we are now as a nation, I think he is probably the right man," said Davies, who earned one England cap during his career.
"Motivation-wise, he is fantastic, the best I have ever worked with. He knows how to get the best out of a player.
"He has a lot of other strengths, too. He surrounds himself with the best people in the best roles, which allows him to plan and get the best out of his squad."
Davies also thinks Allardyce will improve morale in the dressing room.
"The atmosphere he will create will be similar to Wales or Iceland... that spirit, that togetherness, along with the ability that we have a lacked," said Davies.
"I think he has always thrived on being the underdog and I think England are going to go into a lot of games as underdogs in the future, so it probably will suit the way England are at the moment. "
Allardyce is not the only Premier League manager linked with the England job.
Arsenal's Arsene Wenger, Bournemouth's Eddie Howe and Hull City's Steve Bruce have all been touted as possible candidates.
United States boss Jurgen Klinsmann and former England manager Glenn Hoddle have also been linked with the job, but Allardyce is the frontrunner.
The former West Ham, Newcastle, Bolton and Blackburn boss, who has one year left on his Sunderland contract, won a lot of plaudits for keeping the Black Cats in the Premier League last season.
England begin their 2018 World Cup qualifying campaign on 4 September, when they face Slovakia in Trnava.
The two sides met at Euro 2016, battling out a goalless draw in Group B.
Media playback is not supported on this device
The England manager will be chosen by a three-man panel - Gill, FA technical director Dan Ashworth and chief executive Martin Glenn.
They want the next boss to be a strong-minded, tactically savvy manager who will build a clear team identity and help shape the team into a cohesive unit.
Allardyce was previously interviewed for the role in 2006 after Sven-Goran Eriksson left following that year's World Cup.
Eriksson's assistant Steve McClaren got the job instead but he failed to guide England to qualification for the 2008 European Championship.
Here's what Allardyce has since said about the England job:
2009
After missing out to McClaren:
"I should have got it and I really don't know why I didn't. It had to be political for me, rather than my credentials.
"Maybe my external look isn't to everybody's liking. It was the right time and the right job for me but not from the FA's point of view.
"It seems foreign coaches are still all the craze for the top jobs and that is a great shame. I also think that Steve not being successful was a massive blow for British or English managers, because it has put us down a peg or two."
2015
From his autobiography:
"I wanted to do a real knock-your-socks-off interview for the FA, so I put together a PowerPoint which looked at every single detail.
"Nobody but nobody was going to beat it. But then Brian Barwick, the chief executive, told me there were no PowerPoint facilities at the interview venue, so I had to print off hard copies for the panel. So much for the progressive FA.
"I should have got it and, as I'm a better manager now than I was then, I believe I should be in the running whenever it comes round again. That's not vanity or being full of my own importance. My track record entitles me to be considered."
May 2016
Prior to Euro 2016:
"That's gone. For me to be interested in the England job, Roy would probably have to leave at the end of the Euros.
"And would England be interested in me? They say they are looking for an English manager, but will they do it? You've got this, 'what's sexier?' element now, rather than how good you are at doing the job."
Sunderland have given the Football Association (FA) permission to speak to their manager Sam Allardyce about the position of England boss.