Hansjoerg Haber came under fire for comments made last month in which he castigated the government's implementation of the deal.
Mr Haber had been in his post only since October.
The agreement is aimed at halting the mass movement of migrants into Europe.
It revolves around Turkey taking steps to prevent thousands gaining access to Europe in return for Turks gaining visa-free access to the EU bloc.
In comments made to the media on 13 May, Mr Haber, who is from Germany, said: "We have a saying 'Start like a Turk and end like a German. But here it is the other way round'."
His words were seen as deeply derogatory in Turkey, where ambassadors are not expected to disrespect Turkish culture.
On Tuesday, Turkey's EU Minister Omer Celik accused Mr Haber of breaching basic rules of diplomacy by not honouring Turkey's values and its president.
"There are two rules any diplomat should know," Mr Celik told Turkish television while on a visit to Bratislava.
"First is respect for a country's values, and second is respect for that country's president. The ambassador violated these basic rules."
Turkish media reported that the row upset EU foreign policy chief Federica Mogherini, who reportedly put pressure on Mr Haber to resign.
EU officials say he will leave his post on 1 August and a successor will be announced soon.
Turkey's efforts to secure up to â‚¬6bn (Â£4.8bn; $6.8bn) and visa-free travel for its citizens is being held up because EU officials say the government has not met all the demands made by Brussels.
One of the most contentious issues is the EU's demand that Turkey refrain from using its anti-terror laws to arrest academics and journalists.
Prime Minister Binali Yildirim insisted on Tuesday that the government had no plans to compromise on this issue, especially at a time when the military is fighting a bitter war against the outlawed Kurdistan Workers Party (PKK).
Other areas of dispute between Ankara and Brussels include media freedoms and a law that strips Turkish MPs of immunity.
Turkey for its part is concerned over the slow progress of its efforts to join the EU, which dates back to 1987.
The European Commission is on Wednesday due to publish its second report on the implementation of the EU-Turkey agreement, which will include updates on any  progress made in the visa liberalisation dispute.
The EU's top diplomat in Turkey has resigned after a row with the government in Ankara over his criticism of its conduct in a landmark migrant deal.