A small Potter-themed event with stalls and games had been planned for Bearsden Cross, on the outskirts of Glasgow,.
But the Reverend Roddy Hamilton of New Kilpatrick Parish Church said the plans had been "blown out of the water" by the massive interest on Facebook.
The festival has been cancelled  amid concerns that it had got out of hand.
Mr Hamilton, one of the co-ordinator's of the Bearsden Festival, said people had been planning to travel from all over Europe and some had compared it to T in the Park, a music festival that attracts 80,000 a day.
He said it was a small, local event that had been held for five years.
The aim was to develop "better relationships" between the church and the community, he told BBC Scotland's Kaye Adams programme.
Mr Hamilton said: "This was blown out of the water this week by our Facebook post going viral and Potter fans from all over the world being interested in it."
He said the page had reached 250,000 people, with 10,000 confirming that they wanted to attend.
"We got messages from Croatia, Norway and Holland," he said.
"It's amazing what Harry Potter fans are like, they will fly anywhere - broomsticks or otherwise - to get to a Harry Potter festival."
The minister said they were not set up for the level of interest it generated and there were concerns about "safety, parking capacity and licensing implications".
He said: "We were going to do one or two events, a barbeque, a Quidditch match.
"One of the local cafes would decorate themselves as one of the houses in Hogsmead ,where Harry Potter and his friends used to meet, and we would create Butter Beer.
"It was very small scale but it didn't end up being like that."
He said Facebook had been useful to get local interest in previous festivals but the Harry Potter theme sent the post global.
"We just saw the numbers increasing and increasing," Mr Hamilton said.
"It took us three years to get 155 likes on our church Facebook page and three days to get to 10,000 on the Harry Potter page."
Harry Potter superfan Kathryn Burnett, from County Durham, told Call Kaye she had been was planning to go to the festival and was "gutted" it had been cancelled.
"I think I had been tagged on the Facebook page 12 times by friends," she said.
"If it is an Harry Potter event then I'm there.
"I think the fans would love something like this.
"If something on that scale, a huge Harry Potter festival was done, it would be jam-packed. For fans that would be a dream come true."
The Harry Potter Weekend had been due to take place between 23 and 25 June 2017.
Get news from the BBC in your inbox, each weekday morning
The church minister whose Harry Potter festival attracted worldwide interest has said he was taken by "surprise" by the scale of the reaction.