text: Title: Enhance the memory efficiency of loading large models (400B) to prevent out-of-memory errors when using tensor parallelism.
  Body: ### Feature request
  
  Support shredded checkpoint file that matches the process rank for Pytorch distributed model creation and tensor papalism inference.
  
  ### Motivation
  
  When I attempted to test the Llama 405B model with FP8 precision using tensor parallelism (TP = 4), the server, which has 1.5TB of RAM, experienced process termination due to all four processes consuming the entire memory. However, if each process could import the model using a shared weight file and create a model with PyTorch distributed tensor, it would only require 405GB of RAM.
  
  ### Your contribution
  
  I can help to create test cases for this feature.