Question: I have heard that benzene is a dangerous chemical for humans; it causes cancer and so it's use is restricted/regulated. But many organic compounds that we use contain benzene in some form such as oils, foods, even medicines (paracetomol). So is benzene really harmful to people's health?

Answer: 
As Ivan Neretin commented, parts of a compound can never dictate its own properties to the entire molecule's properties. Additionally benzene is not a very effective functional group by itself.

In short, a restricted use is only applied when that certain chemical is exposed by itself. In paracetamol or any other thing including benzene, you will not see any benzene vapor emanating from them or something. It would also be meaningful if benzene containing compounds degradation products included benzene, like in environmental degradation of chlorobenzenes, though this is not a usual case as I see.
Containing a benzene-like (phenyl) group does not make a compound have the same biological activity as benzene itself. Nearly all proteins in your body contain the amino acids phenylalanine and tyrosine, both of which have a phenyl group. Your blood constantly carries these amino acids in their free forms. However, these amino acids are not metabolized by enzymes the same way as benzene and therefore are not metabolized into the epoxide-bearing molecules (described in other answers) responsible for DNA damage.