Question: I googled but there's no answer... I have tried etags but I wonder if there's any more equivalent. Thanks.

Answer: 
A structure that we use on my workplace is having one "External" folder that contains an "Include" and a "Lib" folder for the headers and external Libs. However, as we also use VS, we try to use its "Dependencies" feature as most as possible, removing manual inputs to the linker. That means only projects that are not under the same solution goes to the "External" folder. Also, as we have some third-party libraries specific to some projects, we create folders inside the project folder for these includes and libs. This is how it gets:

```

.\
|-Project1\ --> contains Project1.vcproj
|-Project2\ --> contains Project2.vcproj
| |-Third-Party\
|   |-Include\
|   |-Lib\
|-External\
| |-Include\
| |-Lib\
|-InternalLibrary\ --> contains InternalLibrary.vcproj
|-Solution.sln --> includes the vcproj's and link them as necessary by its dependencies

```

I can't tell if this is the best structure ever, but everything is under source control, we can do night builds just by building the solution, and development goes by building single projects.
There's a fallacy in your statement: "I want to trim the project down to just what's relevant to the program itself."

Believe me, the dependencies are extremely relevant to the program. If you don't have these in source control then you will encounter no end of problems when introducing new team members or when switching to a new workstation.

Even if compile the "non-relevant" libraries, these compiled libraries should go into your source repository.