Question: One of my senior collegues optimizes a function (he is implementing image filtering) by writing in-line assembly. Is that really necessary? Wouldn't modern compiler do that for us? Typically, how much gain do we have by converting C code into assembly? If assembly code really brings lots of benefits, when should we convert C/C++ code into assembly and when should we leave the code as it is, since assembly code is hard to read and maintain.

Answer: 
1. First find an efficient algorithm.
2. Then implement it in clearly readable code.
3. Then evaluate its performance.
4. If your code's performance is inadequate, consider alternative algorithms
5. Repeat steps 3 and 4 until either performance is acceptable or you have exhausted all algorithmic alternatives
6. Drink some coffee.
7. Take a walk.
8. Repeat steps 3 and 4 again some more.
9. Have a beer.
10. Give steps 3 and 4 another few tries.
11. Get some rest
12. Back to 3 and 4.
13. Spend years studying the architecture of the CPU(s) your code will run on
14. Now consider hand-writing some assembly.
Where image processing is concerned I would be cautious either way as it depends on the input data, the algorithm and the compiler. Intel's ICC has a very good parallelizer and vectorizer for generating SSE code, it may be hard to beat by hand in *most general purpose* image processing cases. VCC on the other hand might not do such a good job.
However, I would expect that most benefit could be gained using compiler intrinsics rather than inline assembler.