Question: I thought it would be so because the mass inside the earth would attract us however they would be at a larger distance than masses on the surface of the earth.

For example, consider having a tall building near the object. Would the effective force vector of the object be pointing into the center of the earth or deviated?

Answer: 
The assumption that the force of gravity points to the center of the earth is based upon the assumption that its center of mass is the center of a sphere, I.e., the earth is spherical and of uniform density. 

But since the earth is flatter (not a perfect sphere) there are some deviations in the direction of gravity because of that. Variations at the surface (mountain ranges vs plains near sea level) means it’s mass is not perfectly spherically symmetric and the distance to the center varies slightly. Those factors can theoretically result in very slight deviations in the magnitude and direction of gravity.

Hope this helps.
The force vector would be deviated. That the gravitational force is pointed to the center of the earth is a simplification that assumes the earth is a point mass.
In fact all masses pull on each other, so the mass of your tall building will pull on the object, too, deviating it.
It might even be that the mass of the earth isn't equally distributed - quiet sure - and this causes some (very small) deviation.