Most of the memory is used for storing activations at each layer of your model because these activations are required for computing gradients during the backward pass. This is usually way larger than the size of your model parameters.
nn.DataParallel is used to split your computation among multiple GPUs. Your batch is split among the GPU devices specified.
cudnn.benchmark is for choosing algorithms for performing the layer operations (eg. convolution). It tries to find the fastest algorithm for the model in questions but at the expense of some memory. Setting it to False should reduce the GPU memory needed by a little.