Dug a bit into this code again.  The tensor is created like this:
const auto options = torch::TensorOptions().dtype(torch::kFloat32).device(torch::kCUDA).requires_grad(false);
auto t = torch::from_blob(ptr, {4, 34, 160, 160}, options);

This will eventually segfault when we attempt to dereference data_ptr().  But if instead we use these options:
const auto options = torch::TensorOptions().dtype(torch::kFloat32).requires_grad(false);

…then everything works fine.  Note the removal of .device(torch::kCUDA).
I’ve looked through the github issues looking for segfault related to data_ptr() and torch::kCUDA, and found some things such as libtorch Segmentation Fault: RHEL 7 - easy to reproduce · Issue #19969 · pytorch/pytorch · GitHub 1.  I did attempt to call cudaDeviceSynchronize() just prior to getting data_ptr() but that did not help.
PyTorch is v1.7, cloned from github in early 2021-02, and built manually on a Jetson NX.  It does have the NVIDIA pytorch patch pytorch-1.7-jetpack-4.4.1.patch obtained from the NVIDIA Jetson site along with the instructions on how to build.