Thank you @ptrblck
My code is working only withour DDT (when I comment out the code below)
210         # For distributed training, wrap the model with torch.nn.parallel.DistributedDataParallel.
211         if args.distributed:
212                 model = DDP(model)
213                 # model = DDP(model, device_ids=[args.gpu], output_device=args.gpu)
214                 if args.verbose:
215                         print('Since we are in a distributed setting the model is replicated here in rank {}' .format(args.local_rank))

This is the output without DDT
Singularity> python3.8 -m torch.distributed.launch --nproc_per_node=2 Contrastive_Learning.py --b 128 -v /projects/neurophon/
ERROR: ld.so: object '/soft/buildtools/trackdeps/${LIB}/trackdeps.so' from LD_PRELOAD cannot be preloaded: ignored.
*****************************************
Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
*****************************************
ERROR: ld.so: object '/soft/buildtools/trackdeps/${LIB}/trackdeps.so' from LD_PRELOAD cannot be preloaded: ignored.
ERROR: ld.so: object '/soft/buildtools/trackdeps/${LIB}/trackdeps.so' from LD_PRELOAD cannot be preloaded: ignored.
distributed is True, then rank number 1 is mapped in device number 1
Using device cuda:1 in rank number 1
distributed is True, then rank number 0 is mapped in device number 0
Using device cuda:0 in rank number 0
function_f created from rank 0
function_g created from rank 0
SimCLR_Module created from rank 0
function_f created from rank 1
function_g created from rank 1
SimCLR_Module created from rank 1
pipe1 built by rank number 0 in 3.062119245529175 seconds
Initialating fixation

pipe2 built by rank number 0 in 0.0029039382934570312 seconds
pipe1 built by rank number 1 in 2.993931531906128 seconds
Initialating fixation

pipe2 built by rank number 1 in 0.003136157989501953 seconds
Singularity> exit

This is the output when I comment out all that has to do with DALI and uncomment DDT section
Singularity> python3.8 -m torch.distributed.launch --nproc_per_node=2 Contrastive_Learning.py --b 128 -v /projects/neurophon/
ERROR: ld.so: object '/soft/buildtools/trackdeps/${LIB}/trackdeps.so' from LD_PRELOAD cannot be preloaded: ignored.
*****************************************
Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
*****************************************
ERROR: ld.so: object '/soft/buildtools/trackdeps/${LIB}/trackdeps.so' from LD_PRELOAD cannot be preloaded: ignored.
ERROR: ld.so: object '/soft/buildtools/trackdeps/${LIB}/trackdeps.so' from LD_PRELOAD cannot be preloaded: ignored.
distributed is True, then rank number 1 is mapped in device number 1
Using device cuda:1 in rank number 1
distributed is True, then rank number 0 is mapped in device number 0
Using device cuda:0 in rank number 0
function_f created from rank 0
function_f created from rank 1
function_g created from rank 0
SimCLR_Module created from rank 0
function_g created from rank 1
SimCLR_Module created from rank 1
Traceback (most recent call last):
  File "Contrastive_Learning.py", line 295, in <module>
    main()
  File "Contrastive_Learning.py", line 213, in main
    model = DDP(model, device_ids=[args.gpu], output_device=args.gpu)
  File "/usr/local/lib/python3.8/site-packages/torch/nn/parallel/distributed.py", line 335, in __init__
    self._ddp_init_helper()
  File "/usr/local/lib/python3.8/site-packages/torch/nn/parallel/distributed.py", line 441, in _ddp_init_helper
    self.reducer = dist.Reducer(
RuntimeError: CUDA error: the launch timed out and was terminated
NCCL error in: /pytorch/torch/lib/c10d/../c10d/NCCLUtils.hpp:69, unhandled cuda error, NCCL version 2.4.8
Traceback (most recent call last):
  File "Contrastive_Learning.py", line 295, in <module>
    main()
  File "Contrastive_Learning.py", line 213, in main
    model = DDP(model, device_ids=[args.gpu], output_device=args.gpu)
  File "/usr/local/lib/python3.8/site-packages/torch/nn/parallel/distributed.py", line 335, in __init__
    self._ddp_init_helper()
  File "/usr/local/lib/python3.8/site-packages/torch/nn/parallel/distributed.py", line 441, in _ddp_init_helper
    self.reducer = dist.Reducer(
RuntimeError: CUDA error: the launch timed out and was terminated
NCCL error in: /pytorch/torch/lib/c10d/../c10d/NCCLUtils.hpp:69, unhandled cuda error, NCCL version 2.4.8
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/local/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.8/site-packages/torch/distributed/launch.py", line 261, in <module>
    main()
  File "/usr/local/lib/python3.8/site-packages/torch/distributed/launch.py", line 256, in main
    raise subprocess.CalledProcessError(returncode=process.returncode,
subprocess.CalledProcessError: Command '['/usr/local/bin/python3.8', '-u', 'Contrastive_Learning.py', '--local_rank=1', '--b', '128', '-v', '/projects/neurophon/']' died with <Signals.SIGABRT: 6>.
Singularity> exit

Now let’s go to versions
In the container I instal the following version of dali
# nvidia dali
pip3.8 --no-cache-dir --disable-pip-version-check install --extra-index-url https://developer.download.nvidia.com/compute/redist nvidia-dali-cuda100

Also:
Singularity> cat /etc/centos-release
ERROR: ld.so: object '/soft/buildtools/trackdeps/${LIB}/trackdeps.so' from LD_PRELOAD cannot be preloaded: ignored.
CentOS Linux release 7.8.2003 (Core)


Singularity> python3.8
ERROR: ld.so: object '/soft/buildtools/trackdeps/${LIB}/trackdeps.so' from LD_PRELOAD cannot be preloaded: ignored.
Python 3.8.3 (default, Oct 16 2020, 20:24:57) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-39)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> print(torch.__version__)
1.6.0
>>> 


Singularity> nvidia-smi 
ERROR: ld.so: object '/soft/buildtools/trackdeps/${LIB}/trackdeps.so' from LD_PRELOAD cannot be preloaded: ignored.
Tue Oct 27 12:46:37 2020       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.64.00    Driver Version: 440.64.00    CUDA Version: 10.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla K80           On   | 00000000:07:00.0 Off |                    0 |
| N/A   27C    P8    33W / 149W |     70MiB / 11441MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla K80           On   | 00000000:08:00.0 Off |                    0 |
| N/A   21C    P8    29W / 149W |     69MiB / 11441MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0      1181      G   /usr/bin/Xorg                                 68MiB |
|    1      1181      G   /usr/bin/Xorg                                 68MiB |
+-----------------------------------------------------------------------------+
Singularity>