Pinterest, Engineering, Infrastructure, Open Source.
Training with the same resources & model architecture, but with progressively more complex in trainer data processing, has shown significant throughput decrease. Even if we horizontally scale the training workload through distributed training, it is very challenging to find the right balance