Artificial Intelligence, Machine Learning, Parallel Computing, Diffusion Models, Content Generation.
to 23.2GB. Model parallelism avoids increasing the CPU memory footprint by using lazy_init in the main process, thereby obtaining a meta model with almost no memory footprint. Colossal-AI quantifies and slices the model in the main process, and uses lazy_init in each of the remaining processes to