For  DataParallel  (DP), it can become slow when the model is large, as DP needs to replicate the model in every forward pass.

Ahh that makes sense why that is slower, especially for the models that I have a couple large embeddings.
The lines after with learn.distrib_ctx(): use DDP under the hood as a context manager that handles setting up and tearing down the distributed model. You can find a link to the code here 4, though it is a bit abstracted and a little difficult to understand (at least for me) depending on familiar with the fastai library.
I am guessing there might be an issue with fastai functions/defaults for how it reads number of distributed GPUs available in sagemaker environments.

When using DDP, did you reduce the per-process  batch_size  to  batch_size / world_size ?

Slightly unclear what you mean here. I had the same batch size for each training loop which meant that each GPU in the DP would have been receiving 1/4th the batch size which I was assuming should have been faster.