Question: My wife and I and our adult children have inherited a few acres in the middle of a forest in the mountains of North Carolina and would like to set up a semi-permanent campsite. 

Ideally, this would allow us to camp there with minimum preparation but still have minimal impact on the site and not attract vandalism from passers-by.

Answer: 
Unless you are going to build a sturdy structure, you cannot be confident that you will escape vandalism over the long term whether by homo sapiens or some other species. You will also have to be prepared for normal maintenance. Even a sturdy structure is not absolute proof against vandalism or people breaking in.

Your best option, proof against vandalism, but something of a PIA, is packing everything you need in a closet at home in a highly organized manner, including clothes and things like toothbrushes, and haul it to the site each time. 

You might also want to investigate [roof top tents](https://outdoors.stackexchange.com/questions/10838/roof-top-tents-vs-conventional-tents).

**Addendum:** I should have made my answer clearer when, or shortly after, I posted it. **You cannot make a *semi-permanent* campsite proof against trespassers, squatter, or vandals.** 

As soon as you improve your campsite, passers-by (the OP's phrase) will use it. Some, maybe many, users will be respectful of the site, but eventually someone is going to litter, cut down a tree or otherwise deface or make a mess of your site. At a minimum, they will trample down vegetation and make a path you don't want. All you can do is minimize the hassle of assembling and packing stuff for your visits. Enlisting a neighbor to watch your site will warn you when your site has become a mess. Campsites in parks don't stay clean by themselves!
There is a [great answer by Willeke](https://outdoors.stackexchange.com/a/19236/4079) and a [reasonable answer by ab2](https://outdoors.stackexchange.com/a/19235/4079) in addition to the points they suggested, I have a suggestion.

Find a used travel trailer in your area (e.g., Craigslist) buy it, and move it to the property, and leave it there. This makes your base camp. Leave the door unlocked; if someone wants in, the lock is not going to stop them, and they won't need to break anything to get in. 

Avoid RV's with an engine. They can be difficult to move and or recycle when when you want to upgrade.