Both DropConnect in EfficientNet and PyTorch Dropout seem to divide by (1 - p) to help maintaining the mean (although this isn’t true if there are negative values, e.g. after batchnorm, or swish)
respectively:
  
      github.com
  
  
    lukemelas/EfficientNet-PyTorch/blob/master/efficientnet_pytorch/utils.py#L149 10

        return inputs

    batch_size = inputs.shape[0]
    keep_prob = 1 - p

    # generate binary_tensor mask according to probability (p for 0, 1-p for 1)
    random_tensor = keep_prob
    random_tensor += torch.rand([batch_size, 1, 1, 1], dtype=inputs.dtype, device=inputs.device)
    binary_tensor = torch.floor(random_tensor)

    output = inputs / keep_prob * binary_tensor
    return output


def get_width_and_height_from_size(x):
    """Obtain height and width from x.

    Args:
        x (int, tuple or list): Data size.

    Returns:



  
  
    
    
  
  


  
      github.com
  
  
    pytorch/pytorch/blob/v1.8.1/aten/src/ATen/native/Dropout.cpp#L62 1


  at::Tensor b; // used for alpha_dropout only
  auto noise = feature_dropout ? make_feature_noise(input) : at::empty_like(input, LEGACY_CONTIGUOUS_MEMORY_FORMAT);
  noise.bernoulli_(1 - p);
  if (alpha_dropout) {
    constexpr double alpha = 1.7580993408473766;
    double a = 1. / std::sqrt((alpha * alpha * p + 1) * (1 - p));
    b = noise.add(-1).mul_(alpha * a).add_(alpha * a * p);
    noise.mul_(a);
  } else {
    noise.div_(1 - p);
  }

  if (!alpha_dropout) {
    return multiply<inplace>(input, noise);
  } else {
    return multiply<inplace>(input, noise).add_(b);
  }
}

#define ALIAS_SPECIALIZATION(ALIAS_NAME, IS_FEATURE, IS_ALPHA)                      \



  
  
    
    
  
  


The difference between dropoutNd and dropconnect (as in efficientnet) is that the former drops some channels for each batch sample (1st and 2nd tensor dimensions, batch and channel), but dropconnect drops whole samples all together (only the 1st batch dimension).
So as a late answer, I don’t think the NLP implementation above is equivalent to any of two here. I believe dropout1d would be an approximation to the paper’s algorithm, as it tries to maintain the statistics between training and inference, and drops whole feature channels of each sample.