I don’t think this is possible using a tensor, since e.g. the strides would be hard (impossible) to handle.
Python containers (e.g. list) are quite flexible regarding different types of elements, but you won’t be able to apply e.g. a matrix multiplication on this “array”.
If you are running out of memory, you could have a look at torch.utils.checkpoint to trade compute for memory or e.g. apex/amp for mixed precision training.