your network is very unique and interesting.
One thing that will help us understand the problem better is if you can do the following:
install pytorch from source. Instructions here: https://github.com/pytorch/pytorch#from-source 21
Use the autograd profiler to take a profile summary: http://pytorch.org/docs/master/autograd.html#profiler 139
The autograd profiler is for now only available in master branch, hence the need for install from source.
Post the profile output here, we will look further.