Am I supposed to format when upgrading my mac? I'm on a macbook aluminium late 2008 10.5.8. I'm planning to upgrade it a little bit since many applications doesn't support my system anymore.

At the beginning I was tempted to upgrade to 10.8 but after I read some reviews saying that it's not that good (crushes/need reboot etc) maybe it's better if I just upgrade to 10.6 then reconsider further upgrades in the future.

I just wanted to ask if it's possible to keep all my files and applications while upgrading or if it's mandatory to format and install the new system from scratch.

I'm sure that a clean installation will be better but it would require me much more time in order to backup everything and/or re-install everything I need.
An upgrade in place should be absolutely fine.  I'm sure you can find anecdotal evidence of it crashing for some people, but it's not something that crops on on here more than occasionally.

Note that you cannot jump direct to 10.8 though, sadly (unless you can borrow the install media from someone with a copy) because 10.7 and 10.8 are both sold via the Mac App store, and I think (someone will correct me on this) that this is only available on 10.6.8 systems and later...

If you know someone else with a Mac that has access to the Mac App Store, you can use their to log in and purchase it so that you are properly licensed, but there is no official way to actually download and upgrade from a 10.5 system (although it is relatively trivial if you can get the media).

If you wanted to stage via 10.6 first, that would ease your upgrade to 10.8 (or 10.9 which is due very soon), but you may conversely struggle to find original media for 10.6 (grabbing an original CD from eBay would work so long as the disc is undamaged, but technically it's almost certain that you would not be officially licensed to install it if it was used by the original purchaser to upgrade from 10.5 already, or if it was subsequently used to upgrade to 10.7 or 10.8 - however this is very much a problem for your morals and honesty, because there are no license keys or other media checks in place for Apple to spot this)