Question: I thought it would be so because the mass inside the earth would attract us however they would be at a larger distance than masses on the surface of the earth.

For example, consider having a tall building near the object. Would the effective force vector of the object be pointing into the center of the earth or deviated?

Answer: 
The force vector would only point towards the centre of the Earth if it were spherically symmetric and non-rotating (except perhaps at the equator or poles). There are significant deviations from this and local gravity can deviate by tenths of a degree from a line towards the centre of the Earth.

e.g. see <https://www.wolframalpha.com/widgets/view.jsp?id=e856809e0d522d3153e2e7e8ec263bf2>

I believe that the rotational effects (which are easily predicted) are usually somewhat bigger than the effects caused by local geography.

These deviations were used historically to estimate the [mean density of the Earth](https://en.wikipedia.org/wiki/Schiehallion_experiment) by Neville Maskelyne among others, who used the deviation of a plumbline caused by the mountain Schiehallion.
Geodesists typically distinguish between gravity and gravitation. Gravity is gravitation plus centrifugal acceleration and is measured by the acceleration of an object released at rest with respect to the rotating Earth, as measured from the surface of the rotating Earth.

A plumb bob points "down" -- the direction in which gravity (not just gravitation) pulls the plumb bob. The nominal angular difference between "down" and toward the center of the Earth reaches a maximum of about 0.1924° at about 45° latitude. This nominal angular difference results from the more-or-less ellipsoidal shape of the Earth. "Down" nominally points in the direction of the inward normal to the ellipsoid.

Even on eliminating centrifugal acceleration from consideration, leaving gravitation only, the resultant acceleration only points to the center of the Earth at the equator and the poles. The Earth's rotation results in a bulge of material around the equator, the equatorial bulge. The primary effect of this bulge on gravitation is called "J2". At a constant distance from the center of the Earth, the equatorial bulge makes gravitation stronger than spherical gravitation above the equator and weaker above the poles. The gravitational bulge also results in a component of acceleration that is orthogonal to the direction to the center of the Earth. This effect is greatest at about 45° latitude, where the angle between the nominal gravitation vector and the vector toward the center of the Earth is about 0.9°.

The Earth is not quite an ellipsoid. It has mountains and pockets of material the are more dense or less dense than nominal. (Mountains are for the most part huge pockets of material that are less dense than nominal.) These regions of varying density affect which way gravity (and gravitation) point locally. The angular difference between "down" as observed by a plumb bob and "down" as calculated by assuming an ellipsoidal Earth is called the deflection of the vertical. This deflection is very small, rarely over an arc minute.