2025-04-02_14-52-39 / sft_pretrain_and_pushtohub.log
masani's picture
End of training
2c183d1 verified
[2025-04-02 14:52:59,306][root][INFO] - gcc -pthread -B /dlabscratch1/amani/.conda/envs/LLM-RL/compiler_compat -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /dlabscratch1/amani/.conda/envs/LLM-RL/include -fPIC -O2 -isystem /dlabscratch1/amani/.conda/envs/LLM-RL/include -fPIC -c /tmp/tmpwrka7koq/test.c -o /tmp/tmpwrka7koq/test.o
[2025-04-02 14:52:59,327][root][INFO] - gcc -pthread -B /dlabscratch1/amani/.conda/envs/LLM-RL/compiler_compat /tmp/tmpwrka7koq/test.o -laio -o /tmp/tmpwrka7koq/a.out
[2025-04-02 14:52:59,802][root][INFO] - gcc -pthread -B /dlabscratch1/amani/.conda/envs/LLM-RL/compiler_compat -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /dlabscratch1/amani/.conda/envs/LLM-RL/include -fPIC -O2 -isystem /dlabscratch1/amani/.conda/envs/LLM-RL/include -fPIC -c /tmp/tmpbm8f7_mn/test.c -o /tmp/tmpbm8f7_mn/test.o
[2025-04-02 14:52:59,820][root][INFO] - gcc -pthread -B /dlabscratch1/amani/.conda/envs/LLM-RL/compiler_compat /tmp/tmpbm8f7_mn/test.o -L/usr/local/cuda -L/usr/local/cuda/lib64 -lcufile -o /tmp/tmpbm8f7_mn/a.out
[2025-04-02 14:52:59,881][root][INFO] - gcc -pthread -B /dlabscratch1/amani/.conda/envs/LLM-RL/compiler_compat -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /dlabscratch1/amani/.conda/envs/LLM-RL/include -fPIC -O2 -isystem /dlabscratch1/amani/.conda/envs/LLM-RL/include -fPIC -c /tmp/tmphrk9_0u6/test.c -o /tmp/tmphrk9_0u6/test.o
[2025-04-02 14:52:59,896][root][INFO] - gcc -pthread -B /dlabscratch1/amani/.conda/envs/LLM-RL/compiler_compat /tmp/tmphrk9_0u6/test.o -laio -o /tmp/tmphrk9_0u6/a.out
[2025-04-02 14:53:00,791][__main__][INFO] - *** Starting SFT training at 2025-04-02 14:53:00 ***
[2025-04-02 20:27:41,400][huggingface_hub.utils._http][WARNING] - HTTP Error 500 thrown while requesting PUT https://hf-hub-lfs-us-east-1.s3-accelerate.amazonaws.com/repos/04/d1/04d1d2a0e2109b4ec73350783a4fb45ded670ea39dcbddfbb63e7a2a284adc74/61e708fffd839240a6602b0e299d4fe1aa641dc9e46ee0eb6e6debf6a9380f1e?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=AKIA2JU7TKAQLC2QXPN7%2F20250402%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20250402T202602Z&X-Amz-Expires=86400&X-Amz-Signature=1d268f05d9885d5e9663ee9f5c2256f8a98359e522ec6eb3ff683520070e5bd9&X-Amz-SignedHeaders=host&partNumber=196&uploadId=Oan8ro5WI4HyiYXEpgU3_zT6ehPC3d4ytpYq50cu2GXr_uhmZc_LGcQ2D_7aFPXYJJb1XuQRAW5.rqAmRAY3vbsA3P7HouZFZdy.J5m.ukrWbwDMbZWlgGTiFDKhCKRI&x-id=UploadPart
[2025-04-02 20:27:41,401][huggingface_hub.utils._http][WARNING] - Retrying in 1s [Retry 1/5].