The Institute for Fiscal Studies has updated its calculations, based on the latest official forecasts.
Pro-Union campaigners have welcomed the analysis.
But those backing independence insisted Scotland's finances would be similar to or even stronger than the UK's.
The referendum on Scottish independence will be held on 18 September, when voters will be asked the yes/no question: "Should Scotland be an independent country?
The IFS has produced two new reports on the financial impact of leaving the UK.
The think tank has previously suggested Scotland would see a budget deficit larger than the rest of the UK and would face spending cuts and increased taxes.
In March, however, it said it had revised its more cautious fiscal forecast because the UK economy was growing more quickly than previously thought.
Based on the latest downgraded forecasts on North Sea oil revenues by the independent Office for Budget Responsibility, the IFS has now predicted a slightly weaker position with a fiscal gap of Â£8.6bn in the first year of independence.
By 2019,  when the rest of the UK is due to have a slight budget surplus, the think tank said Scotland would remain with a hole in its finances.
The IFS said: "If an independent Scotland wanted to achieve a sustainable medium and long-term fiscal position, further tax increases and/or spending cuts would likely be needed after independence."
The think tank's report on the fiscal context of the Scottish government's independence white paper added: "The spending cuts and tax rises outlined do not look to be enough to pay for all of the proposed giveaways.
"This does not mean such a package of reforms is infeasible. But, with a background of budget deficits, enacting these measures looks like it would require bigger cuts to other public services or benefits, or other tax rises, if the government of an independent Scotland were to ensure that its public finances were not adversely affected and remained sustainable."
The Scottish government pointed to the fact the IFS calculations were based on declining oil revenues. It said "record investment" in the oil and gas industry would boost future production.
A spokesman added: "Scotland is one of the wealthiest countries in the world, more prosperous per head than France, Japan and the UK, but we need the powers of independence to enable that wealth to be shared and to build a fairer society.
"An independent Scotland's finances in 2016-17 will be similar to, or stronger than, both the UK and the G7 industrialised countries as a whole, and even on the IFS's projections, Scotland's public finance balance sheet in the first year of independence will be healthier than the UK's was in the most recent financial year.
"And as the IFS report notes, Scottish taxpayers paid Â£789 more per head than the UK in 2012-13, demonstrating the huge contribution that Scottish taxpayers make to the UK economy - with more tax per head paid by Scotland than the rest of the UK for every one of the last 33 years."
Unionist politicians welcomed the latest IFS analysis.
Chief Secretary to the UK Treasury Danny Alexander described it as "yet another independent report which leaves the Scottish government increasingly isolated".
He added: "In addition to the CPPR, Citigroup and others, it shows that an independent Scotland would have a larger deficit than the UK, meaning deeper cuts or tax rises than if Scotland stayed part of the UK.
"It highlights the long-term challenge of declining oil revenues, uncosted policies and an ageing population that I set out in our fiscal analysis last week.
"This analysis showed that every person in Scotland is Â£1,400 a year better off as part of the UK."
A think tank has suggested an independent Scotland would face bigger spending cuts and higher tax rises than previously predicted if it is to balance the books.