I am just thinking not to create tensor every time when forward is called OR not pass by value during forward call as i don’t want it to backpropagate.

If you really want to do that, you can just have it as a global in your python file and re-use it.
But I would advise against doing that as globals are quite dangerous in general!
Recreating it in the forward shouldn’t cost you much if you don’t use the function thousands of times every time to forward through your net.

“grad_output” which is 1

That is not true in general, this is actually whatever the grad computed by the other layers is (lower layers).

“grade_input” which is also 1

Is again that depends on the other layers (the ones above you this time).