NLP, Artificial Intelligence, Data Science, Machine Learning, Neural Networks.
size with the number of data-parallel workers, and cannot be made arbitrarily large without affecting the model quality. Tensor parallelism is a broad class of model parallelism techniques that partitions the individual layers of the model across workers. Tensor parallelism reduces the memory