Question: I have heard that benzene is a dangerous chemical for humans; it causes cancer and so it's use is restricted/regulated. But many organic compounds that we use contain benzene in some form such as oils, foods, even medicines (paracetomol). So is benzene really harmful to people's health?

Answer: 
As Ivan Neretin commented, parts of a compound can never dictate its own properties to the entire molecule's properties. Additionally benzene is not a very effective functional group by itself.

In short, a restricted use is only applied when that certain chemical is exposed by itself. In paracetamol or any other thing including benzene, you will not see any benzene vapor emanating from them or something. It would also be meaningful if benzene containing compounds degradation products included benzene, like in environmental degradation of chlorobenzenes, though this is not a usual case as I see.
Benzene is toxic to humans, it can be oxidised by enzymes in the body (might be a P450 enzyme) to form benzene oxide which is an epoxide. This is 7-oxabicyclo[4.1.0]hepta-2,4-diene which can then react with DNA or other molecules in the body. If the epoxide reacts with water to form cyclohexa-3,5-diene-1,2-diol. Then this can be oxidised again to form 7-oxabicyclo[4.1.0]hept-4-ene-2,3-diol which can react with the DNA.

As a result of this DNA damage benzene is able to cause cancer, it is important to keep in mind that many carcinogens are in fact precursors of the carcinogen. They require some metabolic activation to be able to cause cancer.

Another example is hydrazine which is converted in the liver by formaldehyde and oxidation into diazomethane. As a result the liver is damaged.