MSPs voted by 59 to 56 to reject taking forward Anne McTaggart's Transplantation Bill.
Holyrood's health committee backed the aim, but not the detail of the bill.
The government amendment accepted by MSPs said there were "merits" to such a system and pledged a consultation.
The amendment, lodged by public health minister Maureen Watt voiced "serious concerns" about the "practical impact of the specific details" of the bill as it stood, and prevented it from proceeding to the next stage of consideration.
However, Ms Watt did say a "workable" opt-out system should be considered, calling on the government to "commence work in preparation for a detailed consultation" during the next parliament.
The SNP offered members a free vote on the issue, and 11 administration MSPs voted against the government amendment, which won out only with the votes of Conservative and Liberal Democrat members.
Anyone who wants to donate their organs after death currently has to "opt-in" through the donor card scheme.
Ms McTaggart's bill proposed a move to a "soft opt-out" system which would allow parts of a dead adult's body to be used in transplants in the absence of express permission.
It would have still been possible for people to opt-in to organ donation, but the bill would also have given adults the option of appointing someone to make a decision about authorisation on their behalf.
BMA Scotland had called on MSPs to support the general principles of the Transplantation Bill, and MSPs from across the chamber spoke in favour of it during the debate.
SNP MSPs including Kenneth Gibson, Sandra White and Stewart Stevenson spoke in favour of Ms McTaggart's bill, with Mr Gibson calling on members to "stand up and be counted".
While they and a number of their colleagues voted against the government amendment, the votes of 11 Conservative MSPs and one Liberal Democrat ensured the bill will not proceed to the next stage.
Ms Watt said: "While we're very supportive of measures to increase organ availability, in our view the Member's Bill before the Scottish Parliament is seriously flawed and could actually harm organ donation.
"Many of the measures set out in the Bill could make things worse due to legal ambiguities and delays in decision-making processes.
"We have concerns that the proposals around authorised investigating persons (AIPs) and proxies will add significant complexity into the donation pathway, and may lead to potential donors being lost.
"We also have concerns that provisions in relation to adults with incapacity may make it difficult for such adults, or their relatives, to opt-out, leaving them 'locked in' to donation."
The minister said Ms McTaggart's bill had "helped to raise the profile of the debate".
She said the government's consultation would be "an opportunity to seek the views of the public on a range of issues, including soft opt-out, and determine what action the Scottish government should take next to increase the availability of organs".
Ms McTaggart, who is unlikely to be re-elected to Holyrood in May after placing last in Labour's Glasgow list, said the outcome was "deeply disappointing".
She said: "If the government was serious about introducing soft opt-out, then they should have taken the opportunity to table amendments at stage two and stage three, in accordance with parliamentary procedure.
"I find it staggering that over 200 amendments to the Land Reform Bill have already been lodged, yet the SNP government find my relatively short and less complex bill unamendable."

Police called to the property in St George's Hill, Weybridge, at about 10:00 BST by ambulance crews also found a man in his 30s with serious injuries.
Formal identification has not yet taken place, Surrey Police said.
St George's Hill is a private, gated estate with a golf club and tennis club, where six-bedroom homes are listed for about £14m.
Det Ch Insp Jason Taylor said officers believed they knew the identity of the woman and the man, who was taken to hospital for treatment.
He said next-of-kin had been informed.
"The investigation into this tragic incident is still in the very early stages but at this time I believe this to be an isolated incident,"  the detective said.
The 964-acre estate is described by the St George's Hill Residents' Association as having been "designed for driven, motivated and successful high achievers".
The Scottish government said it would consider bringing forward new legislation on an "opt-out" system for organ donation despite MSPs rejecting a Labour member's bill on the topic.
[NEXT_CONCEPT]
A murder inquiry is under way after the body of a woman in her 30s was found at a house in an exclusive part of Surrey.