Maybe this is useful for you?
I implemented a custom backward function for a ternary network (I read the BinaryNet paper that you cited before also), but to do this, I faced similar problems.
Eventually, breaking your function down into two pieces will solve your problems, too? At least in my case, it is working fine.

  
    
    
    Call backward on function inside a backpropagation step autograd
  
  
    To everyone that faces a similar problem, i.e. manipulating gradients in the backward step: I found a working solution for myself now by breaking the problem down into two steps. 
import torch
import torch.nn.init as nnInit
from torch.autograd import Variable


# implementing the TRAINED TERNARY QUANTIZATION layer
# Zhu et al. ICLR 2017
# 1a)   split the method in two functions: the first method TERNARIZES
#       the weights; the second methods just calls the standard convolutions,
#       i.e.…