Would like to custom-build a bike but I'm not sure on what parts to buy Currently I have a 2011 Trek 3700 (no disc brakes) and just this year have gotten into riding mountain bike trails and would like to have a bike better suited for doing that.

I'd like to keep the budget under $1000 (using Trek Mamba as a target) but when I'm looking for parts online, I seem to come across the most expensive options for each part. I'm a fan of Shimano but willing to use other brands if the price is less but the products are still comparable to one another.

As for specifics for what I'm going to be using it on, it'll be public mountain bike trails (off-road, of course) so nothing too demanding but still find a need for disc brakes and given my height, 6'6" roughly, a 29er seems to be the better option and would prefer a hardtail.

Unless a budget of $1000 is just unreasonable for a DIY....

Thanks in advance!
Upgrading parts is likely to be the most expensive way to get the bike you want. Depending where you are, second hand is the way to get a good bike for the lowest $$$, provided you are careful. The next least expensive option is to go for an end of season new bike. In the end, it costs a lot less if you use the proceeds from selling your old bike rather than upgrading it. 

29'ers  (I have one), give quite a different ride to a 26. Try one out to see if you like it, however if you stay hardtail, I think it's a good way to go. If you are a heavy 6'6, consider the 29'r has longer spokes and the wheels are therefore weaker than a 26 (all else the same). As I am under 70kg (150lb), its not something I have to worry about. Also think about the shock - is it stiff enough for your weight, although the only thing that makes a significant difference at this level of bike is more money, and unless you are over 250lb you should be OK (Trek states 136kg max fro their bikes). 

The Mamba looks like a reasonable fit for you in terms of budget and the riding you propose - it will hold up with OK quality parts, I imagine a bit heavier than ideal, but it won't be a dog either. Trek make good bikes and have a good reputation (where I live) for support - which is important.