metadata
tags:
- phosphobot
- gr00t
- replicate
task_categories:
- robotics
Gr00t Model - phospho Training Pipeline
Error Traceback
We faced an issue while training your model.
Training process failed with exit code 1:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/.pyenv/versions/3.11.10/lib/python3.11/site-packages/torch/nn/modules/normalization.py", line 217, in forward
return F.layer_norm(
^^^^^^^^^^^^^
File "/root/.pyenv/versions/3.11.10/lib/python3.11/site-packages/torch/nn/functional.py", line 2900, in layer_norm
return torch.layer_norm(
^^^^^^^^^^^^^^^^^
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 288.00 MiB. GPU 0 has a total capacity of 79.15 GiB of which 209.62 MiB is free. Process 312612 has 78.94 GiB memory in use. Of the allocated memory 78.18 GiB is allocated by PyTorch, and 267.02 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
0%| | 0/2735 [00:13<?, ?it/s]
Training parameters:
- Dataset: lerobot/aloha_static_pro_pencil
- Wandb run URL: None
- Epochs: 20
- Batch size: 64
- Training steps: 2735
More:
๐ Get Started: docs.phospho.ai
๐ค Get your robot: robots.phospho.ai
๐ Explore on Replicate: Replicate