Media playback is not supported on this device
Richard and Peter Chambers, Rob Williams and Chris Bartley won Britain's third rowing medal, with defending champions Denmark taking bronze and Australia fourth.
Media playback is not supported on this device
The Chambers duo became the first British brothers to win an Olympic medal since Greg and Jonny Searle took bronze in the coxless four at the 1996 Atlanta Olympics.
Richard Chambers said: "That was really, really brutal. We were just fighting, fighting through the whole thing to get ourselves back into contention.
"We struggled to keep with the pace of the other crews in the first quarter of the race but we dug our heels in and fought really hard."
Peter Chambers said: "We are gutted. We were unlucky to get a silver with those conditions. We're delighted with silver but we wanted to get gold.
"Fair play to the South Africans, they won that fair and square."
Williams said: "We wanted to win. We have to be happy to get a medal at our home Olympics. It's a shame we didn't win but what can you do? It's a shame."
The British quartet, who took bronze at last year's World Championships, arrived having won at the
They followed up an  where they overhauled Australia in the final 250m, with the fastest time in the semi-finals to progress to the final.
They were slow out of the blocks and were down in fifth as Denmark took an early lead at 500m.
The British boat fought back and were close to level with Denmark and a resurgent South African crew as they entered the final 500m.
But they just missed out on the final push, getting pipped on the line by South Africa.
It was the first Olympic rowing gold for South Africa and the first medal of any kind for the Chambers brothers, Williams and Bartley.

A multi-faith prayer and release of doves is due to take place at 09:15 BST.
Organisers are encouraging attendees to wear or accessorise in "green for Grenfell" in a display of "reverence and respect amidst the revelry".
A "ring of care" will be formed around the tower by police.
More than 80 people are believed to have died when fire engulfed the Grenfell Tower block of flats on 14 June. The carnival route passes near the tower.
Pepe Francis, chairman of the carnival trust, said: "We support the suggestions of the local community and wider carnival community to start this year's carnival earlier.
"This will be the first opportunity for visitors to carnival to pay their respects for those affected by Grenfell."
Hundreds of hand-drawn tributes, flowers and candles will be protected by fencing, while the public have been asked not to take selfies at the site.
A reflection zone will be placed near the burnt high-rise, where performers will lower the volume of their music.
MP Emma Dent Coad, who likened the carnival to a wake, will give a speech ahead of the dove release.
At 15:00 on both days, hundreds of thousands of revellers along the route are expected to pause and observe a minute's silence to mark the tragedy.
A spokesman for Grenfell United, a residents and survivors group, said: "We hope people will follow these requests from us and our friends and family in the wider community.
"They will enable people to pay their respects; be conscious of our vulnerability; and show solidarity for our ongoing demands for housing and full justice, in the creative spirit of carnival."
Steel barriers, concrete blocks and a ban on vehicles will be in place to protect revellers from the threat of terrorism.
Scotland Yard said there was no specific intelligence but security plans had been "thoroughly reviewed" after the Barcelona attack.
Is the fun being sucked out of Notting Hill Carnival?
Rewind: More than 50 years of fun at Carnival

Angela Reakes, 24, of the ACT Meteors in the Women's National Cricket League, was guilty of breaching Cricket Australia's (CA) anti-corruption code.
She pleaded guilty to placing five bets on the Australia v New Zealand final man-of-the-match betting market.
CA integrity unit chief Iain Roy said: "Angela knows she made a bad mistake."
He added: "All elite cricketers are reminded regularly that betting on any form of cricket is strictly prohibited."
Australia won the World Cup final in March, beating New Zealand, with James Faulkner named man-of-the-match.
Great Britain's lightweight men's four won silver as South Africa beat them to gold by just a quarter of a second in a thrilling final at Eton Dorney.
[NEXT_CONCEPT]
Notting Hill Carnival will open with a special ceremony as a "small act of remembrance" following the Grenfell fire tragedy.
[NEXT_CONCEPT]
An Australian cricketer has received a two-year suspended sentence for bets totalling Aus $9 (£4.36) during the 2015 Cricket World Cup.