Explosives lit up the sky and loud bangs were heard as the fire took hold at 05:00 BST on Bitterne Road West.
Homes and properties within 100 metres (328ft) of the fire on the site of the Southern Fireworks Factory and the Flower Factory were evacuated.
No-one was injured. Surrounding roads are expected to remain closed overnight.
More than 70 firefighters in 13 engines and support vehicles helped bring the blaze under control.
Hampshire Fire and Rescue Service group commander Howard Watts said: "It was a challenging scene when we arrived - with all the explosions we couldn't get too close to the fire.
"Our main concern making sure everyone in the area was safe and to make sure no-one was in the building."
At the height of the blaze, a plume of black smoke billowed over the area.
Resident Jill Sainsbury said she heard an "almighty bang" and saw different colours appear in the sky at around 05:00.
She said: "We thought it was coming from the river until we saw a massive plume of smoke coming from the Fireworks Factory near the Flower Factory.
"The fireworks had been going off for at least an hour - it has stopped now - but certainly the plume of smoke is still pouring over the river and the city."
Ryan Gregory, who lives nearby, said he initially thought the bangs were gunshots.
He said: "I looked out of my window to see there was a lot of smoke. Immediately I knew it was from the Firework Factory, but I didn't expect it to go up in the blaze that it did."
The A3024 Bitterne Road West and and Bitterne Bridge have been closed in both directions while firefighters deal with the blaze.
Southampton City Council said it had lifted the toll on the Itchen Bridge until the road reopens.
The authority said the road closures were expected to remain in place overnight
Southern Water also warned the considerable use of hydrants in the area had left some residents with poor pressure or temporary loss of their water supply.
Bitterne Manor Primary School on Quayside Road was also closed.
A rest centre has been set up at the Ascension Centre in Thorold Road for 60 residents who have had to leave their homes.
The council confirmed a licence to store fireworks had been issued to Southern Firework Factory Ltd in September.
A spokesman said the storage conditions at the premises "complied with the relevant conditions".
"At the time of the licence application the premises was purely commercial with no residential accommodation attached. As such the licence permitted 250kg (550lbs) net explosive content to be stored in a steel container secured against unauthorised access," he said.

The 40-year-old German replaces Alex Neil, who left the club in March.
Farke, who spent his playing career in Germany's lower leagues, replaced Huddersfield boss David Wagner at Dortmund's second team in 2015.
Norwich's sporting director Stuart Webber was part of the Terriers' backroom team when Wagner joined.
Edmund Riemer, Farke's assistant at Dortmund, has been appointed assistant head coach at Carrow Road.
"I had a really good time with Borussia Dortmund, but Stuart Webber has told me all about Norwich City, its traditions and history, and I know what a great club it is," Farke told the club website.
"The more I know about Norwich City, the more excited I get. I'm very glad to be here and can't wait to get started."
Norwich finished eighth in the Championship table this season, 10 points outside the play-off places.
A huge blaze broke out at a shop in Southampton, setting off dozens of fireworks stored inside.
[NEXT_CONCEPT]
Championship club Norwich City have appointed Borussia Dortmund reserve-team coach Daniel Farke as their new head coach on a two-year deal.