This is a snippet that works for me from what you described as trying to do:
import torch
from torch.autograd import Variable

x = Variable(torch.randn(10), requires_grad=True)
y = x ** 2
a = Variable(torch.randn(10), requires_grad=True)
b = a ** 2
y[5:] = b[5:]
z = y ** 3
z.sum().backward()

Can you give me a small snippet of the failure case?
I’m wondering if the last layer of your first and second CNN have in-place ReLU or something, which makes them restricted from doing another in-place op on their output. This should be easy to workaound (make the ReLU non in-place).