Question: One of my senior collegues optimizes a function (he is implementing image filtering) by writing in-line assembly. Is that really necessary? Wouldn't modern compiler do that for us? Typically, how much gain do we have by converting C code into assembly? If assembly code really brings lots of benefits, when should we convert C/C++ code into assembly and when should we leave the code as it is, since assembly code is hard to read and maintain.

Answer: 
I'd imagine that for image filtering you might benefit from e.g. the availability of [SIMD](http://en.wikipedia.org/wiki/Streaming_SIMD_Extensions) instructions, but not all compilers can automatically compile your code to use them, and not all the time. So in-line assembly or [intrinsics](http://en.wikipedia.org/wiki/Intrinsic_function) can help with that.
The short answer is no it's not necessary, the longer answer is... well, it depends. Modern compilers do indeed do a very good job of optimizing code, but they don't necessarily have access to all the assumptions a human does when optimizing. Hand coded assembler can beat compiled code, but there is a tradeoff between portability and maintenance.

Assuming that you have already determined this bit of code is a hotspot, the first thing you should do is tweak algorithms, then tweak the C++ code to make it faster (for example unrolling loops), and then tweak compiler flags. As a last resort, if you still can't make it go as fast as you need, consider whether its worth paying the cost of hand-optimizing, given all the future cost you will incur in maintenance and portability.