I don’t believe there is a bug. I believe that some operations can’t be used in-place but the ReLu seems like it can. If the in-place doesn’t screw with autograd for ReLu perhaps the benefit is that it’s slightly faster / less memory since we don’t have to allocate new memory for it. This is just speculation and I’d love to hear from someone more knowledgable.
I for one am going to start using the in-place ReLu though 
Edit: I’ll do a quick experiment when I get to work. -> Okay so I got a 100mb out of 8gb improvement when I used the in-place in my yolov3 model during training. My gpu is gapped at 8gb if anyone is interested
import torch
import torch.nn as nn

x = torch.randn(16, 3, 512, 512).cuda()

conv = nn.Conv2d(3, 32, 3).cuda()
act = nn.ReLU() -> 1115951104 in memory
# act = nn.ReLU(inplace=False) -> 1115951104 in memory
# act = nn.ReLU(inplace=True) -> 584716288 in memory

x = act(conv(x))
print(torch.cuda.max_memory_allocated())