Ok, so I completely wiped it and followed the standard PyTorch install directions from PyTorch versus the nVidia recommended pytorch_jetson_install.sh from Dusty.
The only thing is I added these changes as per recommended by nVidia since NCCL is only for CUDA desktop GPUs:
diff --git a/CMakeLists.txt b/CMakeLists.txt
index f7b24b728..f75f610ed 100644
--- a/CMakeLists.txt
+++ b/CMakeLists.txt
@@ -93,7 +93,7 @@ option(USE_LMDB "Use LMDB" ON)
option(USE_METAL "Use Metal for iOS build" ON)
option(USE_MOBILE_OPENGL "Use OpenGL for mobile code" ON)
option(USE_NATIVE_ARCH "Use -march=native" OFF)
-option(USE_NCCL "Use NCCL" ON)
+option(USE_NCCL "Use NCCL" OFF)
option(USE_SYSTEM_NCCL "Use system-wide NCCL" OFF)
option(USE_NNAPI "Use NNAPI" OFF)
option(USE_NNPACK "Use NNPACK" ON)

diff --git a/setup.py b/setup.py
index 99817f346..e39042b83 100644
--- a/setup.py
+++ b/setup.py
@@ -195,6 +195,7 @@ IS_LINUX = (platform.system() == 'Linux')

BUILD_PYTORCH = check_env_flag('BUILD_PYTORCH')
USE_CUDA_STATIC_LINK = check_env_flag('USE_CUDA_STATIC_LINK')
RERUN_CMAKE = True
+USE_NCCL = False

NUM_JOBS = multiprocessing.cpu_count()
max_jobs = os.getenv("MAX_JOBS")

When I do that, don’t enable TensorRT support, and run the install with Python3 now, it does compile on the fresh install! I do believe I will need the TensorRT support eventually on the TX2, I’ll keep plugging away on that.
For now, however, when trying to import it into a Python shell, I get this:
Python 3.5.2 (default, Nov 23 2017, 16:37:01) 
[GCC 5.4.0 20160609] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/nvidia/pytorch/torch/__init__.py", line 84, in <module>
    from torch._C import *
ImportError: No module named 'torch._C'
>>> exit()