Bhack:

So I suggest you to use an updated version of TF.


Even with 2.6 I see this error:
  Loading requirement: hpcx/2.4.0
2021-08-20 14:23:09.943253: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'li
bcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /cm/
shared/apps/openmpi/cuda/64/3.1.4/lib:/cm/shared/apps/hpcx/2.4.0/sharp/lib:/cm/shared/apps/hpcx/2.4.0/hcoll/lib:/cm/shared/app
s/hpcx/2.4.0/ucx/lib:/cm/shared/apps/cudnn7.6-cuda10.2/7.6.5.32/lib64:/cm/shared/apps/cuda10.2/toolkit/10.2.89/targets/x86_64-
linux/lib:/cm/shared/apps/cuda10.1/toolkit/10.1.243/extras/CUPTI/lib64:/cm/local/apps/cuda/libs/current/lib64:/cm/shared/apps/
cuda10.1/toolkit/10.1.243/targets/x86_64-linux/lib:/cm/local/apps/python3/lib:/cm/shared/apps/gcc5/5.5.0/lib64:/cm/shared/apps
/gcc5/5.5.0/lib32:/cm/shared/apps/gcc5/5.5.0/lib:/cm/shared/apps/slurm/20.11.3/lib64/slurm:/cm/shared/apps/slurm/20.11.3/lib64
:/cm/local/apps/gcc/8.2.0/lib:/cm/local/apps/gcc/8.2.0/lib64:/cm/shared/apps/openmpi/gcc/64/1.10.7/lib64
2021-08-20 14:23:09.943288: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not hav
e a GPU set up on your machine.
2021-08-20 14:24:41.582692: E tensorflow/stream_executor/cuda/cuda_driver.cc:271] failed call to cuInit: CUDA_ERROR_UNKNOWN: u
nknown error
2021-08-20 14:24:41.582920: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:169] retrieving CUDA diagnostic information
for host: node001
2021-08-20 14:24:41.582935: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:176] hostname: node001
2021-08-20 14:24:41.583068: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:200] libcuda reported version is: 460.73.1
2021-08-20 14:24:41.583108: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:204] kernel reported version is: 460.73.1
2021-08-20 14:24:41.583115: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:310] kernel version seems to match DSO: 460.
73.1
2021-08-20 14:24:41.583609: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneA
PI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512
F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2021-08-20 14:24:41.871823: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:185] None of the MLIR Optimization Pass
es are enabled (registered 2)
WARNING: Logging before flag parsing goes to stderr.
W0820 14:24:42.032056 46912496384256 ag_logging.py:146] AutoGraph could not transform <function Model.make_train_function.<loc
als>.train_function at 0x2aab736d7f28> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY
=10`) and attach the full output.
Cause: 'arguments' object has no attribute 'posonlyargs'
To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert