Labour has failed to win back a former stronghold in one by-election and been beaten into fourth in another.
Dai Davies and Trish Law triumphed over Labour
Independent Dai Davies won the Blaenau Gwent seat in south Wales, which Labour lost in 2005 to Independent Peter Law when he overturned its 19,000 majority.
Meanwhile the Tories held Bromley and Chislehurst, in south-east England, but their 2005 majority was slashed from 13,342 votes to 633 by the Lib Dems.
Labour came fourth behind the UK Independence Party.
Labour party chairman Hazel Blears said Blaenau Gwent was a "unique set of circumstances" branding the situation in the Welsh constituency a "family feud".
Dai Davies, Ind: 12,543
(-7,962 from 2005)
Owen Smith, Labour: 10,059
Steffan Lewis, Plaid Cymru: 1,755 (+912)
Amy Kitcher, Liberal Democrat: 1,477 (-4)
Margrit Williams, Conservative: 1,013 (+197)
Alan Hope, Official Monster Raving Loony Party: 318
Turnout: 51.7% (-14.4%)
She said Labour had not had enough time to turn it round from the general election but insisted the result was "coming in the right direction".
She said the Bromley result was an "absolute disaster" for David Cameron's Tories.
"What you've got here is a classic by-election with the Liberal Democrats campaigning really hard, they have a by-election machine," she said.
"And people vote tactically in by-elections, and clearly there was a move here to vote anything but for the Tory party so everybody got behind the candidate that they thought could beat David Cameron's nominee," she told BBC Breakfast.
Conservative leader David Cameron said he was delighted Mr Neill had won even if the margin had been disappointing.
He blamed the Liberal Democrats for fighting a "negative, dirty campaign" that he said was bad for politics as it had put voters off.
"My impression is that very personal, very negative campaigning that the Liberals do - they often do it, they do it a lot and it is damaging and I think it is bad for politics. I think that probably had an effect. Obviously we will have a good look to see what lessons there are to learn," he told BBC News.
Meanwhile, Liberal Democrat leader Sir Menzies Campbell praised his party's near-miss as a "stupendous result" in "one of the safest Conservative seats in the country".
"It shows that there is no confidence in Cameron's Tories in the Conservative heartlands," he said.
Sir Menzies added that the results in both Blaenau Gwent and Bromley were a "complete rejection of the Labour government".
It comes as a YouGov poll for the Daily Telegraph suggested Mr Cameron is more popular than Tony Blair and that 6% more people would prefer a Tory government led by Cameron to a Labour one led by Gordon Brown.
The Blaenau Gwent by-election was called following Mr Law's death in April. Mr Davies had been his election agent.
BROMLEY AND CHISLEHURST
Bob Neill, Conservative: 11,621
(- 11,962 from 2005)
Ben Abbotts, Liberal Democrat: 10,988
Nigel Farage, UKIP: 2,347
Rachel Reeves, Labour: 1,925
Ann Garrett, Green: 811
Paul Winnett, National Front: 476
John Hemming-Clarke, Independent: 442
Steven Uncles, English Democrats: 212
John Cartwright, Monster Raving Looney Party: 132
Nick Hadziannis, Independent: 65
Anne Belsey, Money Reform Party: 33
Turnout: 40.5% (-24.3%)
Mr Law, a former Labour Welsh Assembly member, produced one of the big upsets of last year's general election in winning the seat.
He had left Labour in protest at the introduction of a female-only candidates shortlist. He was diagnosed with a tumour during his election campaign and died this April.
Mr Law won the seat last year with a 9,000-vote majority.
In Thursday's Blaenau Gwent vote, Labour came second, with Owen Smith trailing Mr Davies by just under 2,500 votes.
Mr Smith said there was a need to "reach out to those people who left the party and try to heal some of the rifts that were here".
Bob Neill, who won in Bromley and Chislehurst for the Conservatives, replaces the late Tory MP Eric Forth, who had held the seat since its creation in 1997.
He received 11,621 votes, edging him just ahead of Liberal Democrat Ben Abbotts with 10,988.
Mr Neill was scathing about the Lib Dem campaign
Labour's Rachel Reeves was pushed into fourth behind UKIP candidate, MEP Nigel Farage.
Mr Farage said the result "sends a very loud message that it's time Mr Blair went".
Following his win, Mr Neill accused the Liberal Democrats of fighting their election campaign with the "most vigorous and underhand example of cynical personal abuse" that he had encountered in his 30-year political career.
He told them: "If you sometimes wonder why it is that people in this country are turned off by politics, get a mirror and look at yourselves."
Sir Menzies told the BBC News website members of his party had begun their campaigning before Mr Forth's funeral and before the writ had been moved calling the by-election in Bromley and Chislehurst.
He said he had intervened and stopped them.
He denied Mr Neill's accusation that his party had fought an unscrupulous campaign.
"If there had been any question of impropriety you can be certain I would have stepped in and stopped it," he said.
The Assembly election was won by Trish Law, widow of Mr Law.