if you want gradients to flow through the pre_maps all the way backward, use loss.backward(retain_graph=True).
However, i suspect you will run out of memory and you dont need gradients to flow through pre_maps / hidden_maps. In that case do:
output, hidden_maps = model(input_var, hidden_maps)
hidden_maps = Variable(hidden_maps.data)
loss = F.multilabel_soft_margin_loss(output, target_var)
loss.backward()