he was the one , by the way , who made the family fortune , but i do n't want you to imagine him as the sort of man who slaved over his business , working diligently and watching it grow , prospering slowly over time .