The difference is mainly caused by cuda runtime, that eats hundreds of megabytes once (lazily) initialized; that’s mainly because these is a lot of cuda kernels in torch library. Also, torch memory allocator allocates memory in bigger blocks, so you should also consider cached memory when tracking usage (try torch.cuda.memory_summary()).