what you want to see is that a is a tensor that has four float16 items and b has two float32 items, and they should point to the same memory location as x_gpu, correct?

Correct

If so, I think you have to implement your own code to treat a tensor as different data types (still passing byte type tensor and pass some meta when launch your gpu kernel code?).

There is no custom GPU kernel code. I’m using libtorch because I require access to other C libraries and it’s more convenient to write the entire thing in C++ than a mix of python, C & C++ (since python extensions require C afaik).
The first thing that happens after the data comes out would be a cast to float32 (float16 is only used because the play buffer needs to be large), but it still seem really inconvenient and limiting to have to write an entire kernel to do the cast, when code to cast already exists, but cannot be used because the tensor has the wrong datatype.
The other thing that might fix this is something like tensor::from_blob that uses that as the data pointer. Does anything like that exist?