Buddhism is not fatalism? According to the Buddhism's theory, one man's fate is destined by the past's karma.
Suppose that he faced an unfortunate incident at birth, which is due to past karma. Besides, his genes are inherited from his parents. In other words, his brain is also initiated by past karma. 

In this condition, all his ideas are also affected by the experience he has had, which is determined by past karma. To elaborate more, some are clever, some less, which can again be explained by karma.
But if the conventional human brain can be so strongly determined by the existence of karma, how can you say buddhism is not fatalism?
According to the Buddhism's theory, one man's fate is destined by the past's karma.

This is true that our conditions are determined by past karma, but they are determined by present as well.
A Brahman asked the Buddha in the Cula Kammavibhanga Sutta


  Master Gotama, what is the reason, what is the condition, why inferiority and superiority are met with among human beings, among mankind? For one meets with short-lived and long-lived people, sick and healthy people, ugly and beautiful people, insignificant and influential people, poor and rich people, low-born and high-born people, stupid and wise people. What is the reason, what is the condition, why superiority and inferiority are met with among human beings, among mankind?


The Buddha replies by saying that our conditions are due to our past karma. This; however, is not fatalistic. The idea of karma revolves around the idea that we create our future. 

In the Devadaha Sutta the Buddha refutes two Jainism theories about karma 


The past determines present pleasure and pain, and the present determines future present and pain. 
The belief that you can do nothing in the present to get rid of suffering; all one could do is live with it. 


The Buddha rejected both of those theories that were taught in Jainism. Those two statements sound fatalistic.

The Buddha said that the past and present determine the present; thus, making it not fatalistic. 
Suffering, according to the Buddha, is due to past and present karma as well. 
There is no fatalism in Buddhism. 

The karma you mention in your question seems to be more of Jainist theories of karma than Buddhist theories of karma.