The low-frequency active sonar, approved in 2012, is used to detect enemy submarines.
But it can also harm whales, dolphins and walruses that rely on underwater sound for navigating, catching prey and communicating, according to environmental groups.
The case will now be sent back to a lower court for further consideration.
In 2012, the National Marine Fisheries Service permitted navy sonar use but required it to shut down or delay use when a marine mammal was detected near the ships. Loud sonar pulses were also banned near coastlines and in certain protected waters.
Environmental groups, led by the Natural Resources Defense Council, filed a lawsuit in response, claiming the approval violated the Marine Mammal Protection Act.
The San Francisco federal appeals court ruled that the approval rules granted to the US Navy failed to meet a section of the protection act that required the programme to have "the least practicable adverse impact on marine mammals".
While it said it believed the navy had attempted to follow guidelines, it concluded that the fisheries service "did not give adequate protection to areas of the world's oceans flagged by its own experts as biologically important".
According to experts, the sonar systems used by the navy generate sound waves that can reach 235 decibels - a loud rock band reaches around 130.
These sound waves can travel for hundreds of miles under water and can retain an intensity of 140 decibels as far as 300 miles from their source.
Although direct correlations are hard to find, some scientists believe that the use of sonar has changed the behaviour of whales which have been observed swimming away, sometimes hundreds of miles, rapidly changing their depth and beaching themselves.

The report for 2015 is due out in the next few weeks, so the most recent report we have is the 2014 one.
In the year covered by that report, 1,013 tests were conducted for IPC sports (which include athletics, swimming, powerlifting and skiing) in which prohibited substances were found nine times.
During the year, sanctions were taken against 11 athletes.
If you break that down into the sports involved it is clear that most of the IPC's anti-doping problems come from the sport of powerlifting, which was the source of seven of the positive tests and eight of the sanctions.
The IPC announced in 2014 that it had suspended 13 powerlifters in the previous 14 months. The sport has been targeted with a "Raise the Bar - Say No! to Doping" campaign.
Since then, the IPC has also adopted a new anti-doping code, which came into effect on 1 January 2015.
Paralympic athletes have the same list of banned substances as Olympic athletes and, like them, anyone who requires additional medication for pain or treatment must apply for an exemption.
"It's a common misconception that a lot of Paralympians will be on prescription drugs. That's not the case at all," Nicole Sapstead, UK Anti-Doping chief executive, told the BBC ahead of London 2012.
"Obviously there are athletes with spinal injuries and they need pain relief. But mostly it is the same as the Olympics - things like asthma and diabetes."
Only one athlete was given a doping sanction at the Sochi Paralympic Winter Games in 2014, but since then the McLaren Report for the World Anti-Doping Agency (WADA) found that Russia's sports ministry manipulated urine samples provided by its athletes between 2011 and 2015.
The report identified 27 samples relating to eight Para-sports, five of which are summer sports, including some governed by the IPC.
The IPC also found evidence that samples were swapped during the Sochi Winter Paralympics, adding that it planned to reanalyse every Russian sample given at the event.
So now the Russian team has been banned from this year's Paralympic Games in Rio, and the IPC is wondering if it has more of a doping problem than it previously thought.

They will replace Michael Buble as hosts of next month's show after the Canadian singer pulled out to care for his ill three-year-old son.
The Brit Awards take place at the O2 in London on 22 February.
"The Brits is an iconic show that I've watched and attended many times," said X Factor presenter O'Leary.
Emma, who is currently presenting The Voice UK, said: "To present the Brits is an honour but to be doing it alongside one of my absolute favourites, Mr Dermot O'Leary is just bonkers. Bring on the Brits 2017."
Robbie Williams, Little Mix, The 1975 and Emeli SandÃ© have already been confirmed to perform at the event.
Sonar approved for use by the US Navy broke marine laws, a US appeals court has ruled.
[NEXT_CONCEPT]
The International Paralympic Committee (IPC) gives details of anti-doping activities in its annual reports.
[NEXT_CONCEPT]
Dermot O'Leary and Emma Willis are to present this year's Brit Awards, organisers have announced.