Wightlink began operating larger cross-Solent ferries from its new Lymington terminal in 2009.
The Lymington River Association (LRA) claimed it was harming wildlife habitats.
At a hearing at the Court of Appeal, Lord Justice Sullivan refused it permission to appeal further.
The campaigners had attempted to appeal over a 2011 planning inquiry decision about Wightlink's project to operate larger W-class ships between Lymington and Yarmouth, which had commenced two years earlier.
Wightlink had been allowed to continue running larger ships on Lymington River after promising to "offset" the environmental impact.
Opponents mounted legal action, claiming the ships were creating a bigger wash and were harming habitats in the Lymington River.
But following a public inquiry, the planning inspector agreed that a plan to dredge material from the river to replenish marshland was sufficient.
New Forest's district council and national park authority withdrew their objections to the scheme after it was revealed Natural England had agreed to the plans to replenish the marsh.
The ferry company said the legal action had cost it Â£3.5m in five years.
John Burrows, chief operating officer, said: "While we recognise the rights of individuals to challenge developments on environmental grounds, we believe this case has gone too far.
"It seems to us to be quite wrong that a small group of individuals should be able to impose such a costly legal burden on the UK taxpayer and on our company."
Stephen Akester, of the Lymington River Association, said the decision was "very disappointing".
He said the objections were mounted by "responsible citizens seeking the truth and conservation of the marshes which protect and form an essential part of the character of Lymington".
Campaigners have been refused permission to continue court action against a ferry terminal on the Hampshire coastline.