I am surprised that no one has mentioned this post by Adam Paszke: https://apaszke.github.io/torch-internals.html 203
It gives quite clear introduction into cpu C API. The GPU API is similar if you call your function from python, when the CUDA memory allocation and streams are initialized.