The UK Independence Party has gained its first elected MP, with Douglas Carswell taking the seat of Clacton by 12,404 votes.
Mr Carswell, who defected from the Conservatives, knocked his old party - which enjoyed a 12,068 majority at the 2010 election - into second place.
UKIP leader Nigel Farage said he had "shaken up British politics".
In the night's other by-election, Labour held on to Heywood and Middleton but UKIP slashed its majority to 617.
Mr Farage told BBC Breakfast UKIP could hold the balance of power after next year's general election.
He said UKIP's second place in Heywood and Middleton was "even more significant" than its win in Clacton, saying the party was now the main opposition to Labour in northern cities.
"The whole of British politics has been shaken up", he added.
"Something big is happening here. People want change, they have had enough of career politicians in three parties."
Mr Farage said he expected more Conservative MPs to join UKIP following the defection of Mr Carswell and Mark Reckless, who has triggered a by-election in Rochester and Strood, Kent.
"I think it would be very surprising if more people did not come across", he said, saying he had also spoken to Labour MPs "frustrated that they are not able to change things in British politics".
'Stay true'
Conservative Party Chairman Grant Shapps told BBC Breakfast the result in Clacton was a "wake-up call" saying that while UKIP were a threat to all the Westminster parties, they "cost Conservatives seats", so put Labour leader Ed Miliband "one step closer to Downing Street".
BBC political editor Nick Robinson said the Conservative strategy of saying a UKIP vote helped Labour was "the only way they think they can get those voters back".
The UKIP vote could still "slump back" at the general election, or it could "carry on breaking every prediction and really make a breakthrough", he added.
Labour leader Ed Miliband said: "There won't be a shred of complacency from us as we reach out to all those voters who didn't vote Labour and those who didn't vote at all."
In his acceptance speech Mr Carswell, who gained 21,113 votes - 60% of the total - told UKIP supporters there was "nothing that we cannot achieve".
Addressing the people of his constituency, he said: "I resigned from parliament to face this election because I answer first, foremost and last to you. You are my boss. I will not let you down.
"To my new party I offer these thoughts: humility when we win, modesty when we are proved right. If we speak with passion, let it always be tempered by compassion."
He also addressed the party's prospects on a wider front, saying "we must be a party for all Britain and all Britons: first and second generation as much as every other".
"Our strength must lie in our breadth. If we stay true to that there is nothing that we cannot achieve. Nothing we cannot achieve in Essex and East Anglia, in England and the whole country beyond."
Mr Carswell condemned the other Westminster parties for operating "cosy cartel politics".
Michael Dugher, Labour's vice-chairman, said his party would continue to "expose UKIP for what they are".
"We will take them on in our areas", he said.
Mr Dugher said UKIP had inflicted a "humiliating defeat for David Cameron" in Clacton.
But Labour MP and former Pensions Minister Frank Field MP said: "If last night's vote heralds the start of UKIP's serious assault into Labour's neglected core vote, all bets are off for safer, let alone marginal seats at the next election."
(BBC)