From September, NHS staff and council workers will be among those required to have language skills equivalent to GCSE grade C or above.
Managers will have to test employees can "communicate effectively with the public".
Cabinet minister Matt Hancock said it was to help control immigration.
The new rules, which will form part of the government's immigration bill, will mean every public sector worker in a "customer-facing" role will have to be able to speak at least school-leaver levels of English.
It will include police officers, social workers, teaching staff and assistants and council employees.
The government wants the policy to be rolled out across England, Scotland and Wales and said it would be working with the Scottish and Welsh governments on the details of the plans.
Each organisation will have to decide how to test their staff based on guidance in a code of practice, which the government will produce after a consultation starting in the autumn.
Existing employees who are not fluent enough may be given time to improve.
Doctors working in the UK are already required to have a much higher level of English and are tested by the General Medical Council.
The risk of a healthcare professional not being fluent was highlighted by a mistake made by Dr Daniel Ubani, a German doctor who gave a lethal dose of a painkiller to patient David Gray in 2008.
At the time, as a European citizen, he was able to register to work in the UK without passing a language test but rules were changed in June last year.
Mr Hancock said: "We are controlling immigration for the benefit of all hard-working people.
"We have already introduced tough new language requirements for migrants, now we will introduce new legislation in the forthcoming immigration bill to deliver the commitment made by the prime minister to go further."

The work at Haymarket bus station will include 11 new bus stands at the Charles Street facility.
The city council also is bidding for an additional £3.5m in funding from the Department of Transport.
"The Charles Street station is a disgrace - it is not good enough because it's cramped and inconvenient," Mayor Sir Peter Soulsby said.
"It is just not good enough a modern city like Leicester.
"We have to make sure that travelling by bus to and from Leicester is a realistic and attractive choice."
A council spokesman said the project would "help ease bus congestion and queuing" at the station.
Several commercial buildings will be demolished to increase the size of the revamped station and public access will be improved.
The bus station is a base for 25 local bus services and is used by 20,000 people a day.

At an earlier hearing, Det Sgt Matt Swash, 41, who is based at March police station, admitted accessing unauthorised information about a colleague on a police computer system.
At Cambridge Crown Court, Swash was sentenced to two months in prison, suspended for 12 months.
He remains suspended pending internal police misconduct proceedings.
Cambridgeshire Police said Swash would face no further action over other charges, including 11 counts of sexual assault and two counts of misconduct in public office.

Both teams have to play inter-county matches six days after the Ulster final at Owenbeg and Armagh were reported to want a new date set for the decider.
However, the Orchard county have said the fixture will be fulfilled.
On Saturday, Armagh play Donegal in the Nicky Rackard series and Antrim take on Carlow in the  Christy Ring Cup.
"It is not ideal and we do not have the depth of squad that Antrim have," said Armagh hurling manager Sylvester McConnell.
"It presents us with a dilemma - which competition to prioritise.
"We did not formally request that the provincial hurling decider be deferred, and we will definitely play this Sunday's match."
Antrim have been Ulster champions for 15 years in a row.
People who do not speak fluent English will be barred from public sector jobs which involve working directly with the public, the government has said.
[NEXT_CONCEPT]
A £10.5m revamp is planned to expand facilities at Leicester's "cramped" bus station.
[NEXT_CONCEPT]
A Cambridgeshire police officer has been sentenced for an offence under the Computer Misuse Act.
[NEXT_CONCEPT]
Sunday's Ulster Hurling Championship final between Antrim and Armagh will go ahead as scheduled despite concerns over fixture scheduling.