If you want to operate on raw tensors, and have them wrapped in Variables in a way that ensures connectivity, you have to write a new Function. Otherwise, you have to pass in Variables to your similarity_matrix, but it might be very slow like that.