Q:

How did Newton write his equations?

Once, after a lecture, my professor of differential equations said, that Newton did not use derivatives in his work as we do today. He told us that Newton rather used some series expansions for his equations of motion because it helped him to get better approximation.
Unfortunely I have not been able to find any written referense of it, is there any? Or maybe someone knows how that series looked like?

A:

This question involves two rather different points. It involves Newton's practices for notating not only derivatives, but also the functions (or whatever else) he used to indicate the values of those derivatives.  In cases where we might now write dy/dx = {some-function} , the question involves Newton's practices for expressing both the left and the right sides.  'Equations', as we now notate them, were only one among several ways in which he wrote down relationships of this kind, as can be seen from the following examples and their sources.   
The comment given by Michael E2 shows one of the ways (fluxional notation) in which Newton indicated a derivative for which we might now write dy/dx .  Beyond that, Newton also used other ways to indicate derivatives or rates of change, not all of them related to the notation of fluxions, and some of them not involving symbols at all.
One notable type of example can be found e.g. in Proposition 34 (Book 3) of the Principia. Here, Newton undertook 'to find the horary variation of the inclination of the Moon's orbit to the plane of the ecliptic'. Newton gave a rather complicated geometrical specification for the quantity of this variation: it amounts, when translated, to a triple product of sines/cosines. Newton then went on (in Proposition 35) to integrate (a simplified version of) this quantity geometrically with respect to the time, to produce a result showing the periodic variation of the actual inclination of the orbit at any time (strictly, its difference from the mean inclination, which plays the role here of the arbitary constant in the integration). 
From all of that, especially the integration, it is clear that Newton's 'horary variation of the inclination' meant what we could now  write as di/dt , where i is the inclination and t the time. P-S Laplace later treated it so. Thus, when Laplace considered this same passage in Newton, he 'translated' Newton's geometric expression for the value of the 'horary variation' into a trigonometrical formula (see P-S Laplace, Traite de Mecanique Celeste, tome 5 (1825), livre XVI, ch.2, art.3 at p.375, discussing the inclination from p.379 onwards; at (https://books.google.com/books?&id=UdZGAQAAMAAJ)). Laplace then developed its integral (which he did with wider generality than Newton's treatment, which had involved some simplification before the integration, but that doesn't alter the main point of interest here). 
The example so far given shows Newton using, for the quantity of the rate of change in which he was interested, a geometric definition (which was translatable -- and later translated -- into trigonometric notation). In other examples he used algebraic series, but Newton's uses of algebraic series appear to stem primarily from an earlier part of his career.  When Newton did use series, his notation for the terms was sometimes not too different from what might be written now, except for details such as aaa for $a^3$ and so on. See for example The Mathematical Papers of Isaac Newton, volume II: 1667-1670, (ed.) D T Whiteside (1968), (https://books.google.com/books?id=AQ3tveOwseoC), where page 206 et seq. give Latin & English versions of Newton's 'On analysis by equations infinite in number of terms' and show some of his practices for writing down such series.
In 'Correspondence of Isaac Newton' vol.1 at p.52 (24 Dec 1670, John Collins to James Gregory) Collins reported, among other things obtained from Newton, Newton's series for the arcsine.  A longer description of how Newton obtained this (by term-by-term integration of the series obtained by binomial expansion of 1 / $\sqrt(1-x^2)$ ) is given for example (but with modernized notation) in 'The Calculus Gallery: Masterpieces from Newton to Lebesgue', by William Dunham (2005), at p.17.  (Newton's purpose in using the series does not seem, at least in this case, to have been "to get better approximation": rather it was to convert an expression that was effectively intractable as it stood, into a readily-integrable form.)  
(John Collins was a well-known mathematical 'intelligencer' of Newton's time, who saw a number of Newton's (often algebraic) inventions and reported them to his correspondents. Collins later understood that the prospects seemed poor for getting any more mathematics, especially no doubt in algebraic form, from Newton, because Newton (and Barrow) were then "beginning to thinke mathematicall speculations to grow at least nice and dry, if not somewhat barren" (Isaac Newton Corresp vol 1 p.355 (19 Oct 1675, John Collins to James Gregory)). The letter turned out to be prophetic, in that Collins never afterwards heard anything more of mathematics from Newton, and Newton turned more towards the language of geometry to express his mathematical investigations. The reasons for Newton's (reportedly) growing disaffection from the kind of mathematics that he had engaged in up to that time perhaps deserve fuller investigation.)

