Solved by Deep_Shankar_Pandey in post #3 
              
              
                For this problem, shared tensors can be used. 
I asked the same question on stack overflow and the solution works. 
Stack overflow Answer 
Hope this helps others facing similar problem.