They said the higher end of the market was stalling because of the new land and buildings transaction tax (LBTT).
The rate change has resulted in a tax cut on sales under £325,000, but they have increased above that sum.
Representatives from the property sector made their views known in evidence to MSPs at Holyrood.
Members of the finance committee have been examining the LBTT changes which came into being in April this year.
In a paper submitted by the Scottish Fiscal Commission to the committee estimated revenues under the new tax of between £203.9m to £243.9m, once the effect of forestalling - people bringing transactions forward to avoid the new tax - was taken into account.
That compares with the Scottish government's forecast for LBTT revenues of £235m for the year 2015-16, which did not include forestalling.
Under LBTT a rate of 5% is levied on the proportion of a property's price between £250,000 and £325,000, with a 10% rate applying between £325,000 and £750,000 and 12% above £750,000.
The Scottish Property Federation (SPF) wanted the 5% rate threshold to be raised to £500,000 and for the 12% rate to be abolished.
Chairman Chris Stewart said the federation's figures showed a 22.5% decline in sales of properties over £400,000 between May to September 2014 and the same period in 2015, with a 54% reduction in sales of homes costing more than £1m.
Sales to international buyers in the highest bracket were also down from 27% last year to 18% in 2015.
Mr Stewart told the committee: "This is not about feeling sorry for people in that part of the market, it's about allowing the market to function properly.
"Clearly the tax structure, that 12%, is putting people off... it's stopping people from investing in Scotland."
Similar concerns were raised by Philip Hogg, chief executive of Homes for Scotland, which represents the home building industry.
He said: "The impact of the tax at the top tier levels, the ones where the higher tax bands come in, has been quite significant and we're hearing anecdotally a number of our members stating that sales have either been stalled or have not gone through."
Mr Hogg pointed to evidence that buyers at the top end of the market were opting to "spruce up" their homes instead of moving.
Homes for Scotland was not calling for immediate change to the regime but wanted it kept under "close review" with a view to adjusting or tweaking the system if current trends continued.
Kennedy Foster, policy consultant at the Council of Mortgage Lenders, pointed to growth in the mortgage lending market in Scotland, particularly for first-time buyers, but cautioned this could be due to other factors such as low interest rates and a competitive mortgage market.
He said figures for the quarter ending September show first-time buyer numbers up to 8,500 with the value of loans granted for house purchase at £920m.
Prof Campbell Leith, from the Scottish Fiscal Commission, told MSPs: "The out-turn data is significantly below what you would expect given the seasonality you would normally expect in the data.
"This may be because of a temporary forestalling effect or it may be that the change in the tax regime has permanently subdued certain parts of the market and this will continue indefinitely."
Property experts want a review of Scotland's stamp duty replacement as analysis shows it could be down by more than £30m in its first year.