Question: One of my senior collegues optimizes a function (he is implementing image filtering) by writing in-line assembly. Is that really necessary? Wouldn't modern compiler do that for us? Typically, how much gain do we have by converting C code into assembly? If assembly code really brings lots of benefits, when should we convert C/C++ code into assembly and when should we leave the code as it is, since assembly code is hard to read and maintain.

Answer: 
1. First find an efficient algorithm.
2. Then implement it in clearly readable code.
3. Then evaluate its performance.
4. If your code's performance is inadequate, consider alternative algorithms
5. Repeat steps 3 and 4 until either performance is acceptable or you have exhausted all algorithmic alternatives
6. Drink some coffee.
7. Take a walk.
8. Repeat steps 3 and 4 again some more.
9. Have a beer.
10. Give steps 3 and 4 another few tries.
11. Get some rest
12. Back to 3 and 4.
13. Spend years studying the architecture of the CPU(s) your code will run on
14. Now consider hand-writing some assembly.
The short answer is no it's not necessary, the longer answer is... well, it depends. Modern compilers do indeed do a very good job of optimizing code, but they don't necessarily have access to all the assumptions a human does when optimizing. Hand coded assembler can beat compiled code, but there is a tradeoff between portability and maintenance.

Assuming that you have already determined this bit of code is a hotspot, the first thing you should do is tweak algorithms, then tweak the C++ code to make it faster (for example unrolling loops), and then tweak compiler flags. As a last resort, if you still can't make it go as fast as you need, consider whether its worth paying the cost of hand-optimizing, given all the future cost you will incur in maintenance and portability.