Sorry I should’ve posted more of the logs. The CUDA diagnostic does appear to find CUDA. Just not the GPU.
2021-08-20 15:21:38.393015: E tensorflow/stream_executor/cuda/cuda_driver.cc:271] failed call to cuInit: CUDA_ERROR_UNKNOWN: u
nknown error
2021-08-20 15:21:38.393070: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:169] retrieving CUDA diagnostic information
for host: node001
2021-08-20 15:21:38.393081: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:176] hostname: node001
2021-08-20 15:21:38.393208: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:200] libcuda reported version is: 460.73.1
2021-08-20 15:21:38.393248: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:204] kernel reported version is: 460.73.1
2021-08-20 15:21:38.393256: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:310] kernel version seems to match DSO: 460.
73.1
2021-08-20 15:29:06.834136: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneA
PI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 AVX512
F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2021-08-20 15:29:07.343075: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:185] None of the MLIR Optimization Pass
es are enabled (registered 2)
WARNING: Logging before flag parsing goes to stderr.
W0820 15:29:07.578475 46912496383040 ag_logging.py:146] AutoGraph could not transform <function Model.make_train_function.<loc
als>.train_function at 0x2aab74dc1840> and will run it as-is.
Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY
=10`) and attach the full output.
Cause: 'arguments' object has no attribute 'posonlyargs'
To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert
done A stocks
Model: "sequential"
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
lstm (LSTM)                  (None, 64)                16896
_________________________________________________________________
dense (Dense)                (None, 1)                 65

Is that error Cause: 'arguments' object has no attribute 'posonlyargs' just a re herring?