When running with >>>b utils.cpp:8 I see a missing file error:
Breakpoint 2, THPPointer<THLongStorage>::free (this=0x7fffffffba10) at /opt/conda/conda-bld/pytorch_1512378360668/work/torch/csrc/generic/utils.cpp:13
13      /opt/conda/conda-bld/pytorch_1512378360668/work/torch/csrc/generic/utils.cpp: No such file or directory.

Not sure if this is anything meaningful.
(pytorch0.3.0_py2) thomasbalestri@markable02:~/PycharmProjects/pytorch-detect-to-track$ gdb python
GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from python...done.
(gdb) catch throw
Catchpoint 1 (throw)
(gdb) b utils.cpp:8
No source file named utils.cpp.
Make breakpoint pending on future shared library load? (y or [n]) y
Breakpoint 2 (utils.cpp:8) pending.
(gdb) run ./tools/trainval_net.py --weight data/pretrained_models/res101.pth --imdb imagenet_vid_train --imdbval imagenet_vid_val  --iters 100000 --cfg experiments/cfgs/res101.yml --net res101  --set ANCHOR_SCALES [8,16,32] ANCHOR_RATIOS [0.5,1.0,2.0]
Starting program: /home/thomasbalestri/anaconda3/envs/pytorch0.3.0_py2/bin/python ./tools/trainval_net.py --weight data/pretrained_models/res101.pth --imdb imagenet_vid_train --imdbval imagenet_vid_val  --iters 100000 --cfg experiments/cfgs/res101.yml --net res101  --set ANCHOR_SCALES [8,16,32] ANCHOR_RATIOS [0.5,1.0,2.0]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Called with args:
Namespace(cfg_file='experiments/cfgs/res101.yml', imdb_name='imagenet_vid_train', imdbval_name='imagenet_vid_val', max_iters=100000, net='res101', set_cfgs=['ANCHOR_SCALES', '[8,16,32]', 'ANCHOR_RATIOS', '[0.5,1.0,2.0]'], tag=None, weight='data/pretrained_models/res101.pth')
Using config:
{'ANCHOR_RATIOS': [0.5, 1.0, 2.0],
 'ANCHOR_SCALES': [8, 16, 32],
 'CLASS_AGNOSTIC': True,
 'DATA_DIR': '/home/thomasbalestri/PycharmProjects/pytorch-detect-to-track/data',
 'DENSENET': {'FIXED_BLOCKS': 0},
 'EXP_DIR': 'res101',
 'MATLAB': 'matlab',
 'MOBILENET': {'DEPTH_MULTIPLIER': 1.0,
               'FIXED_LAYERS': 5,
               'REGU_DEPTH': False,
               'WEIGHT_DECAY': 4e-05},
 'PIXEL_MEANS': array([[[ 102.9801,  115.9465,  122.7717]]]),
 'POOLING_MODE': 'roi',
 'POOLING_SIZE': 7,
 'RESNET': {'FIXED_BLOCKS': 1, 'MAX_POOL': False},
 'RNG_SEED': 3,
 'ROOT_DIR': '/home/thomasbalestri/PycharmProjects/pytorch-detect-to-track',
 'TEST': {'BBOX_REG': True,
          'HAS_RPN': True,
          'MAX_SIZE': 1000,
          'MODE': 'nms',
          'NMS': 0.3,
          'PROPOSAL_METHOD': 'gt',
          'RPN_NMS_THRESH': 0.7,
          'RPN_POST_NMS_TOP_N': 300,
          'RPN_PRE_NMS_TOP_N': 6000,
          'RPN_TOP_N': 5000,
          'SCALES': [600],
          'SVM': False},
 'TRAIN': {'ASPECT_GROUPING': False,
           'BATCH_SIZE': 256,
           'BBOX_INSIDE_WEIGHTS': [1.0, 1.0, 1.0, 1.0],
           'BBOX_NORMALIZE_MEANS': [0.0, 0.0, 0.0, 0.0],
           'BBOX_NORMALIZE_STDS': [0.1, 0.1, 0.2, 0.2],
           'BBOX_NORMALIZE_TARGETS': True,
           'BBOX_NORMALIZE_TARGETS_PRECOMPUTED': True,
           'BBOX_REG': True,
           'BBOX_THRESH': 0.5,
           'BG_THRESH_HI': 0.5,
           'BG_THRESH_LO': 0.0,
           'BIAS_DECAY': False,
           'DISPLAY': 20,
           'DOUBLE_BIAS': False,
           'FG_FRACTION': 0.25,
           'FG_THRESH': 0.5,
           'GAMMA': 0.1,
           'HAS_RPN': True,
           'IMS_PER_BATCH': 1,
           'LEARNING_RATE': 0.0005,
           'MAX_SIZE': 1000,
           'MOMENTUM': 0.9,
           'PROPOSAL_METHOD': 'gt',
           'RPN_BATCHSIZE': 256,
           'RPN_BBOX_INSIDE_WEIGHTS': [1.0, 1.0, 1.0, 1.0],
           'RPN_CLOBBER_POSITIVES': False,
           'RPN_FG_FRACTION': 0.5,
           'RPN_NEGATIVE_OVERLAP': 0.3,
           'RPN_NMS_THRESH': 0.7,
           'RPN_POSITIVE_OVERLAP': 0.7,
           'RPN_POSITIVE_WEIGHT': -1.0,
           'RPN_POST_NMS_TOP_N': 2000,
           'RPN_PRE_NMS_TOP_N': 12000,
           'SCALES': [600],
           'SNAPSHOT_ITERS': 5000,
           'SNAPSHOT_KEPT': 3,
           'SNAPSHOT_PREFIX': 'res101_faster_rcnn',
           'STEPSIZE': [70000, 140000, 190000, 240000, 1100000, 1160000],
           'SUMMARY_INTERVAL': 180,
           'TRUNCATED': False,
           'USE_ALL_GT': True,
           'USE_FLIPPED': False,
           'USE_GT': False,
           'WEIGHT_DECAY': 0.0001},
 'USE_GPU_NMS': True}
Number of classes: 31
Loaded dataset `imagenet_vidtrain` for training
Set proposal method: gt
Preparing training data...
imagenet_vidtrain gt roidb loaded from /home/thomasbalestri/PycharmProjects/pytorch-detect-to-track/data/cache/imagenet_vidtrain_gt_roidb.pkl
done
Number of classes: 31
38121 roidb entries
Output will be saved to `/home/thomasbalestri/PycharmProjects/pytorch-detect-to-track/output/res101/imagenet_vidtrain/default`
TensorFlow summaries will be saved to `/home/thomasbalestri/PycharmProjects/pytorch-detect-to-track/tensorboard/res101/imagenet_vidtrain/default`
Number of classes: 31
Loaded dataset `imagenet_vidval` for training
Set proposal method: gt
Preparing training data...
imagenet_vidval gt roidb loaded from /home/thomasbalestri/PycharmProjects/pytorch-detect-to-track/data/cache/imagenet_vidval_gt_roidb.pkl
done
Number of classes: 31
5515 validation roidb entries
Filtered 1064 roidb entries: 38121 -> 37057
Filtered 103 roidb entries: 5515 -> 5412
Pairs in roidb: 32943
Pairs in roidb: 4825
Solving...

Breakpoint 2, THPPointer<THLongStorage>::free (this=0x7fffffffba10) at /opt/conda/conda-bld/pytorch_1512378360668/work/torch/csrc/generic/utils.cpp:13
13      /opt/conda/conda-bld/pytorch_1512378360668/work/torch/csrc/generic/utils.cpp: No such file or directory.
(gdb)