From pytorch/c10/cuda/CUDACachingAllocator.cpp 7, I found this:

Allocations are associated with a stream. Once freed, blocks can be re-allocated on the same stream, but not on any other stream.

And in code 2, it actually follows this rule.
I can’t figure out why this limitation exists. As far as I know, CUDA doesn’t forbid different cuda streams to access the memory allocated by cudaMalloc. And this limitation will disallow some free blocks allocated by one cuda stream to be reused by other cuda streams.
Example:
import torch

a = torch.rand((12, 12)).cuda()   # allocated by default cuda stream
print(torch.cuda.memory_cached())

torch.cuda.synchronize()
s = torch.cuda.Stream()
with torch.cuda.stream(s):
    b = torch.rand((12, 12)).cuda()    # allocated by other cuda stream

torch.cuda.synchronize()
print(torch.cuda.memory_cached())

output is: 2097152 and 4194304.
import torch

a = torch.rand((12, 12)).cuda()   # allocated by default cuda stream
print(torch.cuda.memory_cached())

b = torch.rand((12, 12)).cuda()   # allocated by default cuda stream
print(torch.cuda.memory_cached())

output is: 2097152 and 2097152.
Environment: PyTorch 1.3, cuda 10.1