kleingeo:

However, when setting gradient_as_bucket_view to either True or False, I am seeing about 2GB of additional overhead per card.


What is the 2GB additional overhead in comparison to? Are you comparing to when you don’t use DDP at all?
cc @Yanli_Zhao