Question: Some times when I start a file download (direct not p2p) I get download speeds way more than what was promised to me, but then it gradually decreases to a more (un)acceptable speed. 

Is it the file hosting server slowing the speed as a measure to DOS attack or is it my ISP realizing its mistake late ?  

And also why is the decline gradual and not abrupt ?

Answer: 
Your ISP probably has a "boost". So, when you start the download it boosts your speed way up for the first couple of megs (this is common). Or the site you are connecting to could be doing the same thing.
It is possible that the server itself is throttling it. Some servers may prefer to throttle a client's connection except for the first few bytes (or KB, MB). This is more usual in FTP servers though. In this way clients downloading a small file has better user experience, and clients downloading a big file will not slow the server down too much.