gr00t Model - phospho Training Pipeline

Error Traceback

We faced an issue while training your model.

Traceback (most recent call last):
  File "/root/src/helper.py", line 166, in predict
    trainer.train(timeout_seconds=timeout_seconds)
  File "/root/phosphobot/am/gr00t.py", line 1145, in train
    asyncio.run(
  File "/opt/conda/lib/python3.11/asyncio/runners.py", line 190, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/asyncio/base_events.py", line 654, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/root/phosphobot/am/gr00t.py", line 995, in run_gr00t_training
    raise RuntimeError(error_msg)
RuntimeError: Training process failed with exit code 2:
2025-05-23 06:08:30.172792: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2025-05-23 06:08:30.173940: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2025-05-23 06:08:30.181252: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2025-05-23 06:08:31.811331: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
╭─ Unrecognized options ──────────────────────────╮
│ Unrecognized options: --validation-dataset-path │
│ ─────────────────────────────────────────────── │
│ For full helptext, run gr00t_finetune.py --help │
╰─────────────────────────────────────────────────╯

Training parameters:

Dataset: PLB/mkia-3
Wandb run URL: None
Epochs: 6
Batch size: 64
Training steps: None

📖 Get Started: docs.phospho.ai

🤖 Get your robot: robots.phospho.ai