In her resignation letter she said her decision would enable her party to have a "real discussion" about its future.
Senior party members had "questioned my place in this new phase", she said.
Earlier, the 56-year-old said some Labour MPs were "dinosaurs" who failed to see "Scotland has changed forever".
Ms Lamont had wanted more autonomy for the party in Scotland and significant new powers for the Scottish Parliament.
But in an interview with the Daily Record she accused Westminster colleagues of treating Scotland like a "branch office".
Labour has 41 MPs in Scotland who will fight for their Westminster seats at next May's General Election. The party also has 38 MSPs in Edinburgh's Holyrood parliament and they will seek re-election in 2016.
In her resignation letter to Scottish Labour chairman Jamie Glackin Ms Lamont said the referendum had "opened a new chapter in the debate about the future of the Scottish Labour Party".
She added: "In order that we can have the real discussion about how we take Scottish Labour forward, I believe it would be best if I took  myself out of the equation and stepped down as leader."
She described Labour's challenge in Scotland as "serious", but added "I strongly believe that the Labour Party is not only our best chance of achieving a better, fairer Scotland, it's our only chance".
There are two versions of this story.  In one version, Scottish Labour's departing leader, Johann Lamont, accuses her Westminster colleagues - and, by implication, Ed Miliband - of undermining her and failing to grasp how much Scotland has changed.  She says some of her MP colleagues are "dinosaurs".
The alternative version is that Johann Lamont wasn't up to the job - that she failed to counter the SNP, that she failed to modernise the party sufficiently to cope with a new Scotland where people are no longer prepared to back Labour as a duty, that she failed to attract new talent who might freshen up the party's portrait for the electorate in 2016, when Holyrood goes to the polls.
Reacting to news of the resignation, SNP deputy leader Nicola Sturgeon said that Ms Lamont "carries my personal best wishes".
But she added: "There is no question that her shock resignation reveals Labour to be in complete meltdown in Scotland."
Ms Sturgeon said: "The scale of the infighting between Scottish Labour and Labour at Westminster is exposed for all to see."
The timetable for choosing a new leader will be set out soon.
In the meantime, deputy leader Anas Sarwar is in charge and an MSP will be chosen to stand in for Ms Lamont at Holyrood.
Whoever replaces her will become Scottish Labour's seventh leader since the Scottish Parliament was established 15 years ago.
Ms Lamont, a former English teacher who joined the Labour Party as a teenager, has represented the Glasgow Pollok constituency at Holyrood since 1999.
She took over as the party's Scottish leader in the aftermath of the 2011 Scottish Parliament election, when Labour suffered a crushing defeat at the hands of the SNP.
In the weeks after the independence referendum, she said she wanted to lead Labour to victory in the next Holyrood election in 2016 and become first minister.
Labour leader Ed Miliband said she deserved "significant credit" for the successful "No" vote in the Scottish referendum campaign.
He added: "She campaigned the length and breadth of Scotland making the case for social justice within the United Kingdom.
"She has led the Scottish Labour Party with determination. I know she will continue to serve her constituents.
"Having elected a new leader, I believe the party will show the same will and determination it did in the referendum campaign to help us to victory in the general election of 2015 and the Scottish elections of 2016."
Former prime minister Gordon Brown has constantly been linked to the job, as has fellow MP Jim Murphy, who also played a prominent role in the referendum campaign.
Mr Brown said he was sorry to hear that Ms Lamont had resigned.
He added; "She brought determination, compassion and a down-to-earth approach to the leadership and deserves great credit for taking on the challenge after 2011.
"I wish her well in the future."
Senior Labour sources have indicated that both the former Prime minister, Gordon Brown and the former Scottish Secretary , Jim Murphy, will be approached to see if they are willing to lead the Labour Party in Scotland.
If the two men refuse another scenario being canvassed is for a joint ticket between the MP Anas Sarwar and the MSP Kezia Dugdale.
Mr Sarwar would be the leader and would seek to improve relations between Westminster and Holyrood. Ms Dugdale  would lead the group at Holyrood. One Labour source described this as the "ideal option".
Others believe Mr Murphy should be encouraged to stand. One figure said that if "Jim doesn't stand it will be a disaster", but added he believed the shadow cabinet member could be persuaded.
Scottish Labour leader Johann Lamont has resigned saying senior party members had "questioned" her role amid the "serious challenges" faced by the party after September's referendum.