Rescuers earlier ended efforts to find the body of Jeffrey Bush, 36, whose bedroom was engulfed by the 6m (20ft) hole on as he slept on Thursday night.
Sinkholes are common in parts of Florida, although most are quite small.
Experts are trying to ascertain how far the underground cavity reaches and whether more homes are at risk.
All homes near to the house in Seffner - a suburb of 8,000 people 25km (15 miles) east of central Tampa - have been evacuated.
Officials have created a 100ft-wide safety zone fearing the top of the sinkhole is growing.
The house covers most of the hole, and a local official said crews would try to move the building forward to enable surviving family members to retrieve their belongings from it.
"We don't know, in fact, whether it will collapse or whether it will hold up,'' said Hillsborough County Administrator Mike Merrill.
On Sunday, the demolition crews left standing only a few walls of the house.
They were proceeding as carefully as possible to salvage at least some of the home's contents for five survivors.
A Bible, family photos and a woman's purse were among the items saved.
The demolition is expected to be completed on Monday and experts would then examine the hole.
Just before the crews started their work, 35-year-old Jeremy Bush, who had tried to save his brother, was seen praying outside the house.
Five people escaped unharmed, after hearing a loud crash and Jeffrey Bush screaming as the earth crumbled below the house.
Jeremy Bush said he jumped into the chasm to dig his brother out but failed. He was later rescued by police.
Rescuers then used listening devices and cameras at the scene, but found no signs of life.
"There's no way of possible survival," said fire rescue spokeswoman Jessica Damico on Saturday.
Mr Bush told reporters that someone had visited the four-bedroom house some weeks ago to check for sinkholes and other issues but found nothing wrong.
"And a couple of months later, my brother dies. In a sinkhole," he said.
Florida state law requires home insurers to provide coverage against sinkholes.
Experts say much of Florida sits on a system of limestone caverns which are subject to water erosion, causing them to collapse.

The 20-year-old France international was fined and suspended until Monday for missing Thursday's training, but the ban has been extended indefinitely.
Dortmund met with Barcelona's representatives but the Spanish club did not meet their valuation.
The German club say that Dembele moving to Barca is "not currently likely".
Dortmund director of football Michael Zorc said: "Our focus now is on preparation for the Bundesliga opening match in Wolfsburg [on 19 August]. Dembele, of course, has the opportunity to complete an individual training session outside the group."
Dortmund signed Dembele on a five-year deal from Rennes in May 2016.

Cheshire Police released the statistic as they launched a campaign to urge male victims to report abuse.
Officers said they were raising the issue in the run-up to Valentine's Day, "when relationships are at the forefront of everyone's mind".
Information about help will be in GP surgeries, community centres, shops and sports centres.
Assistant Chief Constable Phil Thompson said: "Domestic abuse knows no boundaries and it can happen to anyone at any time regardless of gender, age and sexuality.
"Men in particular find it hard to speak out if they are being abused, but we want to reassure anyone who may be suffering at the hands of another, that domestic abuse is an issue that we take extremely seriously - whatever time of the year.
"Traditionally the emphasis has been on female victims of abuse but it is important to highlight the fact that many victims of domestic abuse are also men.
"By bringing this important issue to the front of people's minds we hope that it will encourage any men to have the confidence in contacting us if they want to end the cycle of abuse."

The Al-Qadi family saw their side go behind to Jamie Devitt's penalty when Tom Parkes was penalised for handball.
But Rory Gaffney equalised with a 12-yard volley after away goalkeeper Barry Roche's unconvincing attempts to deal with a Danny Leadbitter cross.
And substitute Billy Bodin's close-range header earned victory late on.
Rovers go sixth, five points adrift of the top three, while Morecambe are in 17th place, 11 points clear of the relegation zone.
Demolition crews have partially razed the home of a man presumed dead after he was swallowed up by a sinkhole in Florida.
[NEXT_CONCEPT]
Borussia Dortmund have suspended forward Ousmane Dembele "until further notice", three days after the club rejected a bid from Barcelona.
[NEXT_CONCEPT]
One in five victims of domestic violence in Cheshire is male, police have revealed.
[NEXT_CONCEPT]
Bristol Rovers celebrated their first match under their new Jordanian owners with a victory over Morecambe that takes them up to fifth in League Two.