He said they should follow the example of London Challenge,  the now defunct Labour policy which improved London schools, and get outstanding schools and heads working very closely with the less successful.
For decades, London secondary schools had some of the worst exam results in the country.
Sir Michael was a teacher in the capital in the 1970-90s. He told BBC Newsnight that if someone had told him then that London would be doing much better than the rest of England, he would not have believed it.
He said the improvement was down to London Challenge - there was the "political will" to make schools better, adding that London Challenge bypassed local authorities.
Little Ilford School in Newham is one school where results have improved dramatically over the last 15 years.
In 1997, just 16% of its students got five GCSES at grades A-C, the league table measure then. Last year, 71% passed at least 5 GCSES at grades A*-C including English and Maths.
Over half of the school's pupils are eligible for free school meals, and 88% speak English as an additional language.
According to the acting headteacher, Katy Episcopo, that change is down to improved teaching, and certain specific initiatives.
They have Study Plus classes, which help students who are behind in reading and writing, and that has had a significant impact. They work closely with pupils' parents. They build confidence and they support aspiration.
Although funding for London Challenge was cut two years ago, results have kept rising. It seems the improvement has developed its own momentum.
Not only are more students passing GCSEs, but more are achieving higher grades too. Temoore Zulfiqar, who is aged 16, has already taken three GCSEs, in maths and science. He got two A* grades and one A.
He wants to go to Oxford or Cambridge and thanks to his school, he has already visited both several times.
Students at Little Ilford see no bar to their ambitions. Tanvir, who is 15 years old, told me he wanted to be Britain's first Muslim prime minister.
As London has rapidly improved, some other authorities have not kept pace. Barnsley in South Yorkshire has some of the worst GCSE results in England.
Simon Barber, headteacher of the brand-new Holy Trinity school, is determined to improve standards. Although almost all his pupils speak English as a first language, and only 16% were eligible for free school meals last year, 53% of pupils got five GCSES at grades A* to C, including English and maths, last year.
He told me he was keen to learn from other schools, especially those in London. He took a group of children to Newham last summer, to see the London sights, and to visit another school.
Some of the pupils who had gone on the visit told me the atmosphere was different at the Newham school - the children were more focused on learning.
"If you didn't look out the window, you'd never know it was a rough area," one young teenager told me. "The children looked so smart and ready to learn."
Mr Barber told me it was hard to overcome Barnsley's industrial past. The town was a mining community, but now unemployment is high, and he said it was hard to raise aspiration.
Sir Michael Wilshaw told BBC Newsnight that comparing Barnsley and Newham shows that "children are children, are children - what makes the difference is the culture of the school".
Ofsted is now turning its attention to local authorities. If it finds them failing, their role could be taken over by another body, possibly another local authority, or an academy chain.

Eleven men - two posthumously - were given the Légion d'Honneur at a ceremony on the navy helicopter carrier Dixmude at HM Naval Base Devonport.
They fought at the D-Day Landings more than 70 years ago.
Plymouth's honorary French consul said the event on board the warship was "quite exceptional" and would never be repeated.
It was the second time France had held such a presentation on a warship, with a similar ceremony staged on board FS Aquitaine in Glasgow in 2015.
The men who were honoured at Sunday's ceremony are aged between 90 and 97 years old.
Gillian Garrlick collected the medal on behalf of her late father, Charles Dixon, from Plymouth.
"My father had actually received it through the post back in October, so he did know he'd received it but was waiting for an official presentation.
"Unfortunately he died on 30th December so didn't get to be here today, but it was an honour to receive it for him."
Sir Michael Wilshaw, Chief Inspector of Schools in England and head of education watchdog Ofsted, has told BBC Newsnight that schools and local authorities up and down the country should learn from London and "draw a line" below underperformance.
[NEXT_CONCEPT]
D-Day veterans have received France's highest military decoration on a French warship in Plymouth.