Prof Gerry Holtham said plans for an advisory group of business leaders and experts were "probably not sufficient".
He claimed it would be a "big ask" for First Minister Carwyn Jones to add Brexit to his list of responsibilities.
The economist said a dedicated Brexit ministry would have the staff and resources to undertake the job.
Prof Holtham, who previously led an inquiry into Welsh Government funding, said the advisory group was a good idea to help with "very complicated" negotiations, claiming both the Welsh and UK governments were "short on experience and expertise".
But the Cardiff Metropolitan University professor said the panel would not be enough.
"Brexit will influence a lot of different ministries within the Welsh Government," he told BBC Wales, naming agriculture and industry as key areas.
"All of these will have to work together and make their input," he said.
"The Welsh Government hasn't been great in the past at that kind of cross-departmental co-ordination.
"It may help to have somebody who is responsible for pulling it together and conducting the negotiations."
New Prime Minister Theresa May has appointed David Davis as secretary of state responsible for Brexit in the UK government, with former Welsh Secretary David Jones as a junior minister working with him.
Mr Holtham said the first minister may want to lead the Brexit response from Wales himself, but asked: "Does he have the staff back-up to actually carry that out?
"If there were a department doing that, there would be civil servants seconded to it and it would have the resources to undertake the job, as well as having a minister responsible for seeing that all the information was brought together and co-ordinated in the right way.
"It's a big ask for the first minister to do that as well as all the other things that he has to do."
In July, Mr Jones told the BBC's Sunday Politics Wales: "There can't be a minister for Brexit.
"I have a limited number of ministers I can appoint - I can't appoint somebody just to do that.
"The reality is we can do this anyway... not just me but the cabinet and other parties as well."
Reacting to Mr Holtham's comments, a Welsh Government spokesman said: "The first minister is chairing a cabinet sub-committee focused exclusively on Brexit issues, bringing together the departments directly linked to EU business.
"In addition the cabinet secretary for finance and local government will chair an EU advisory group drawing together expertise and sectorial interests from across Wales.
"Throughout this process the Welsh Government will be fighting vigorously for Wales' interests."

Having gone behind through Macauley Gillesphey's free-kick after Jason Kennedy had been hauled down, Orient overturned the deficit.
Jay Simpson found space to level after half-time, firing into the left corner.
Ollie Palmer then streaked clear to put Orient ahead, before Raynes headed in from Alex Gillead's cross to level.
The goal denied Kevin Nolan's visitors the chance to go fifth in the table, as they remain seventh, while Carlisle are three points behind in 11th.
Leyton Orient player-manager Kevin Nolan told BBC Radio London:
"You've got to respect the point, especially when you come to a place like this.
"In the end we probably could have snatched it, but if we would have I think (Carlisle manager) Keith Curle would have been equally disappointed.
"I'm delighted with the response. Since I've come in we've taken 16 points out of eight games so it's fantastic form and we've got to take positives out of it because we're all disappointed at the moment."
Wales needs a dedicated minister for Brexit to co-ordinate its response to leaving the EU, a former Welsh Government advisor has said.
[NEXT_CONCEPT]
Michael Raynes' late equaliser earned Carlisle United a key point in a 2-2 draw with fellow League Two promotion hopefuls Leyton Orient.