I think what this means is that PyTorch and Chainer are smarter than “add everything to a global giant list” in that they store a graph of operations and do differentiate between paths where they need to backprop through and stuff they don’t need - for example if you are finetuning a convnet by just replacing the final layer, only the graph for this last bit from final layer to loss will be saved and backpropagated through.
Also, you can use the “selective recording” (my unscientific language) for neat things like differentiating implicit functions 23 and trading compute for memory 20.
Quite likely, the nuances of no tape vs. advanced, modern interpretation of “tape” are as opaque to laypersons like me as they are clear to experts in the field, but one would perhaps expect that in 2017 you know a trick or two about great data structures and thread safety and stuff beyond what Wengert had in 1964.
Best regards
Thomas