I think the main worry you have with using Variables everywere is the overhead it could imply compared to use pure Tensors directly? This has been looked into in details and in the current master branch, the overhead of using a Variable (with requires_grad=True or torch.no_grad()) is negligible.

Actually I was only worrying about the requires_grad=False case since the inference based models I’m working on only needs tensors and convolutions. I just want to make sure that when I turn off the requires_grad option I can treat the variables as tensors and loop however many times without thinking about the additional structures. maybe “Premature optimization is the root of all evil.” 

Did I miss another worry you had?

Thanks a lot for all of these comments and explanations. Thanks to @SimonW, too.  I think I have a much better idea and turning off the gradient is a solution for now. The new release of PyTorch sounds really exciting!