Question: Some times when I start a file download (direct not p2p) I get download speeds way more than what was promised to me, but then it gradually decreases to a more (un)acceptable speed. 

Is it the file hosting server slowing the speed as a measure to DOS attack or is it my ISP realizing its mistake late ?  

And also why is the decline gradual and not abrupt ?

Answer: 
The calculation of download speed is prone to several issues that result in it giving a relatively inaccurate report at various times. The beginning of the download is perhaps most inaccurate.

When you click the download link, depending on your browser and your settings you'll have to go through a process choosing where the file is to go on your computer. While you're working on this, your browser is actually downloading the file in the background to temporary storage. The download speed calculator will normally start calculating from when it first opens, which is after the file may have been downloading for a little while, which means it's got a whole of data to account for in it's calculation, but only a little bit of time, so the speed appears very, very high. As the time the download window is open comes closer to the amount of time the download has been actually been running, the download speed numbers become more and more accurate.

You can test this by starting a download and letting the file save dialog prompt stay on your screen for a minute, allowing a large amount of data to download, then once you hit save you'll probably see some pretty impressive download speeds that have absolutely no bearing on reality.

ISP Boost can also affect the speed numbers, as Opsin notes in his response, as can different download types and server types.

Your ISP is not "realizing it's mistake" and you'll find, very clearly printed in all the small text of your agreement with the ISP, that they only promise "up to" the speed you're paying for. Depending on your ISP and connection type, you're sharing throughput with many other subscribers (if it's cable, you're sharing with everybody watching Chuck episodes after dinner too) and this will affect how much effective throughput you've got.

If your ISP does not throttle BitTorrent, running a well seeded Torrent can get you the closest to your actual throughput of any type of download. Also, mid-day downloads of very large files from large sites like Microsoft.com will be a reasonably accurate throughput, as their hosting servers are pretty fast and well configured.
It is possible that the server itself is throttling it. Some servers may prefer to throttle a client's connection except for the first few bytes (or KB, MB). This is more usual in FTP servers though. In this way clients downloading a small file has better user experience, and clients downloading a big file will not slow the server down too much.