It took the Tapolca seat with a majority of just under 300 votes.
Jobbik looks set to present a major challenge to the governing Fidesz party in parliamentary elections due in three years' time.
Correspondents say the party has softened its rhetoric in recent years and has gained support from mainstream voters.
Lajos Rig's margin of victory was narrow, but his win was symbolically enormous, says the BBC's Nick Thorpe in Budapest.
"The mood in Hungary is for a change of government and with Jobbik Hungary finally has a force to change the government,'' party leader Gabor Vona told supporters in Tapolca.
It re-enforces their reputation as the most successful nationalist party in Europe and is another setback for Hungary's Prime Minister Viktor Orban.
"There are times when the ball hits the post," said Mr Orban, a keen footballer, on his Facebook page as he admitted defeat.
Fidesz easily won the last national and European elections in 2014, when Jobbik took around 20% of the vote.
But since then the governing party has seen its reputation falter, with allegations of corruption, and opposition to measures like a planned internet tax and limits on Sunday trading.
Fidesz lost its two-thirds parliamentary majority in February after another by-election defeat.

The students began "feeling unwell" at Llanwern High School, in Newport, at about 11.40 BST on Thursday.
One 14-year-old girl, five girls aged 13, two girls aged 12 and an 11-year-old boy were taken to Royal Gwent Hospital.
Laboratory tests are under way to determine what the substance was.
Police said not all the pupils required hospital treatment but were taken as a precaution. Eight of the nine have now been discharged.
Gwent Police said inquiries were ongoing to find out the exact circumstances of what happened and where the substance was obtained.
The Welsh Ambulance Service sent three rapid response cars, three emergency ambulances and an urgent care service vehicle to the school.
A council spokesman said on Thursday: "Our immediate priority is the well-being of the pupils affected.
"The school will be carrying out an investigation and will be working with the police into how this incident happened."

The roof of the house on Bashurst Hill in Itchingfield was spotted alight at 00:30 BST.
West Sussex Fire & Rescue sent four engines to fight the blaze and all occupants were accounted for.
Sussex Police said the cause of the blaze was unknown but there was nothing to suggest it was deliberate.

The National Stadium - or Bird's Nest - was used at the 2008 Olympics.
"When you plan someone's career like Joshua you tend to plan in advance," said Hearn, who saw Joshua stop Charles Martin last month to win the title.
"I think we're two years from saying we can sell out the Bird's Nest in China."
Hearn is the father of Joshua's promoter Eddie, and chairman of the fighter's Matchroom stable.
He identified China's 2008 Olympic silver medallist Zhang Zhilei, who has built an unbeaten professional record of eight fights since basing himself in Las Vegas, as a potential opponent for 26-year-old undefeated Joshua.
"Sometimes these things don't happen because people get beaten, but our job is about creating these kinds of moments," he added.
Joshua will make the first defence of his title against American Dominic Breazeale in London next month.

20 October 2016 Last updated at 14:40 BST
In a short speech, Prof Hawking said: "The rise of powerful AI will be either the best, or the worst thing, ever to happen to humanity. We do not yet know which."
Read more: Stephen Hawking - will AI kill or save humankind?
The far-right Jobbik party in Hungary has won its first ever individual constituency seat in parliament.
[NEXT_CONCEPT]
A 14-year-old girl remains in hospital after nine pupils were taken ill at school after reportedly taking an unknown substance.
[NEXT_CONCEPT]
One person has been taken to hospital after a serious fire gutted a house in West Sussex during the early hours on Saturday.
[NEXT_CONCEPT]
Great Britain's IBF world heavyweight champion Anthony Joshua could make a defence of his title at Beijing's National Stadium, according to promoter Barry Hearn.
[NEXT_CONCEPT]
Prof Stephen Hawking has discussed the potential pitfalls of artificial intelligence, at an event in Cambridge marking the opening of the Centre for the Future of Intelligence.