The fundamental difference is that DyNet’s graphs are global objects held by the singleton autograd engine. In PyTorch (and Chainer) the graphs are attached to the variables that are involved in them and (as Soumith demonstrated) go out of scope when those variables do.