The Briton saved two match points to win 4-6 7-6 (7-4) 7-5 before an angry exchange with the Latvian at the net.
Ostapenko claimed her racquet slipped from her hand in the second-set tie-break and was let off with a warning.
Broady disagreed saying her opponent should be "straight disqualified".
"You'll see on the replay, there's no way it was accidental. She threw the racquet and it hit the ball boy," British number three Broady told the umpire in an on-court exchange.
WTA rules state that "players shall not at any time physically abuse any official, opponent, spectator or other person...for the purposes of this rule, physical abuse is the unauthorised touching of an official, opponent and spectator or other person".
Players can be disqualified for incidents by the umpire and fined up to $10,000 (£6,830) for each violation.
Former British number one Tim Henman was disqualified from Wimbledon in 1995 after accidentally hitting a ball girl with a ball struck in frustration, while David Nalbandian was thrown out of the Queens final in 2012 after kicking out at an advertising hoarding, cutting the shin of a line judge.
World number 88 Ostapenko, ranked 38 places higher than Broady, was allowed to continue after a tour supervisor was called to court, with Broady appearing to be near tears over the incident.
The Briton knuckled down in the third set, reeling off six successive games to snatch victory before becoming involved in an argument following the post-match handshakes.
"Jelena commented that my behaviour was terrible, which I thought was a bit out of order considering the events that had gone on in the match," said Broady of the row.
Broady, who beat former world number one Ana Ivanovic in the first round, will now play American Sloane Stephens.

Dorset and Wiltshire Fire and Rescue worked with Mendip Cave Rescue to free them from the former stone mines in Box after being called at 02:50 BST.
Martin Grass, from Mendip Cave Rescue, who helped rescue the seven, in their late teens and early 20s, said they were "cold and inadequately equipped".
It comes after another group had to be rescued from the same caves on Sunday.
Nobody in either group was injured but the fire service is warning people "to think twice before visiting the former stone mines".
Mr Grass said: "With these two most recent rescues, the groups were woefully ill-equipped for what they were trying to do, and they were very lucky that they found a place of relative safety where there was a phone signal."
After rescuing another group of three men on Sunday evening, the fire service tweeted saying that signs, to help people get out, had been painted over.
Darren Nixon, from Dorset and Wiltshire Fire and Rescue said: "Fortunately, nobody was injured on these two latest occasions, and the groups were OK to stay where they were until the cave rescue team arrived.
"However, with a bit more planning, and a bit more thought, they could have avoided multiple fire crews and volunteers being mobilised.
"Having fun is fine, but please think twice about your plans and whether you have done everything possible to be safe."
In July last year, a group of teenage boys had to be rescued from Box mines after getting stuck 100ft underground while paying the smartphone game Pokemon Go, where users search real-life locations for digital creatures.
Naomi Broady is into the ASB Classic third round after a controversial match against Jelena Ostapenko, who escaped disqualification for appearing to throw a racquet towards a ball boy.
[NEXT_CONCEPT]
Firefighters have rescued a group of people trapped in a Wiltshire cave for the second time in a week.